Hi, i have to create web API to send huge number of records from our oracle database view to use it in one entity which responsible to generate reports etc. From differents sites so they need to take our data every period time i need to know which type web
service i have to use and i need sample or link to start learn i hope it is clear
Regards
Webapi is not really designed for large datasets. You will need to stream the output, and update the timeouts. You might want to look at gRPC which is a better approach
You design your api to send data in page by page format. This means the client will also send the
page no and
no of records per page when making the request to the API.
The API's job will be to do
custom paging on it's end and get that data which the client is asking for. This is what APIs of large corporations employ.
Helping you always. Don't forget to click "Mark as Answer" on the post that helped you.
♠ ASP.NET Core Tutorials → Start from the Beginning and become an Expert in 30 days time ♠
For large amount of data it is suggested to implement batch jobs, using SSIS or some other ETL tool and store all resultant data in one place. Then use that data to generate reports. You can run that batch Job Every 30 min or so that suits your business
needs :)
Thanks Alot for your replay can you tell more about SSIS Or ETL if you have references and also you know this service will be used by other site need our data as centerlize department
You can place your final consolidated data generated from SSIS/ETL as need by others at one central location. This data can be in the form of CSV available at one endpoint. Does that make things easy for you ?
None
0 Points
4 Posts
Web api big data
Jul 17, 2020 11:32 PM|hilal_w|LINK
Regards
All-Star
56394 Points
14756 Posts
Re: Web api big data
Jul 18, 2020 03:11 PM|bruce (sqlwork.com)|LINK
Webapi is not really designed for large datasets. You will need to stream the output, and update the timeouts. You might want to look at gRPC which is a better approach
https://grpc.io/docs/languages/csharp/quickstart/
note: if you need to support legacy systems it may not be an option.
Participant
1161 Points
880 Posts
Re: Web api big data
Jul 22, 2020 05:38 AM|yogyogi|LINK
Large amount of data from web api ?
You design your api to send data in page by page format. This means the client will also send the page no and no of records per page when making the request to the API.
The API's job will be to do custom paging on it's end and get that data which the client is asking for. This is what APIs of large corporations employ.
♠ ASP.NET Core Tutorials → Start from the Beginning and become an Expert in 30 days time ♠
Member
30 Points
30 Posts
Re: Web api big data
Jul 28, 2020 11:53 AM|vsetia|LINK
HI,
For large amount of data it is suggested to implement batch jobs, using SSIS or some other ETL tool and store all resultant data in one place. Then use that data to generate reports. You can run that batch Job Every 30 min or so that suits your business needs :)
None
0 Points
4 Posts
Re: Web api big data
Jul 28, 2020 12:27 PM|hilal_w|LINK
Member
30 Points
30 Posts
Re: Web api big data
Jul 28, 2020 02:33 PM|vsetia|LINK
You can refer to online tutorials here
https://docs.microsoft.com/en-us/sql/integration-services/sql-server-integration-services?view=sql-server-ver15.
You can read everything from development to deployment.
You can watch YouTube tutorials and there are several courses available out there :)
Member
30 Points
30 Posts
Re: Web api big data
Jul 28, 2020 03:18 PM|vsetia|LINK
You can place your final consolidated data generated from SSIS/ETL as need by others at one central location. This data can be in the form of CSV available at one endpoint. Does that make things easy for you ?
Member
30 Points
30 Posts
Re: Web api big data
Jul 29, 2020 10:43 AM|vsetia|LINK
Is there anything else for which i can help you ?