I have some applications that produce reports of information about various things in networks, such as folders or Active Directory objects - in some cases people working at huge companies need to run these programs against literally millions of folders or AD objects, so the amount of data that gets returned can be massive. As the application is only 32 bit (due to some calls to native Windows APIs that I haven't got working correctly on x64 systems yet) it can only access 4 GB of memory at most so even on an x64 system with tons of RAM in some cases people are hitting this limit and the program throws an Out Of Memory exception.
My only workaround for this issue so far has been to add options to write the results directly to CSV file, so that I can use a FileStream to write the results as they are found rather than adding them to a list that is stored in memory (the items are added to a list in memory normally so that they can be displayed to the user and then they can choose to export to CSV or Excel or HTML file from there). But this FileStream method has its drawbacks - for example the results can't be sorted at all because they're being written out to file as soon as they are created so there's no point where each individual item can be compared against all other items, plus I don't like the fact that the user can't see the results in the GUI first and see if they even want to export them.
I can't help thinking there must be a way to make this actually work and have all the data visible in the GUI without it all being in memory at once, by paging the data to/from disk as needed. I know of other programs that must do this, but I don't know of any .NET programs that do this. I know it happens in the background anyway what with the way virtual memory gets paged to/from the Windows page file as required, but that doesn't help avoid the limit of only 4 GB of virtual memory being addressable to 32 bit programs.
I know I could easily use a FileStream to write the data to my own file, writing each entry to disk as it is found by the program, and then just read "small chunks" in at a time - but my problem is how to make that actually work with a GUI. I mean lets say my in-memory list only has half the actual results loaded at the moment, and the user scrolls down my TreeView (or ListView if they're using the alternate view option in the program) or they choose to re sort a column, how do I make that work on ALL of the data when I can't load all of the data into memory at once? How would the scroll bar even work in a vaguely "normal" way at all considering the Tree/ListView won't ever actually have all of the data in it? I'm using WPF for the GUI (just in case someone suggests using some TreeView/ListView events that are specific to WinForms)
I'm wondering if this is something that is too low level for .NET? If not perhaps someone knows if there is actually some way to do this that is going to result in a decent user experience
Oh and inb4 dunfiddlin tells me I don't actually want to do this ;)
My only workaround for this issue so far has been to add options to write the results directly to CSV file, so that I can use a FileStream to write the results as they are found rather than adding them to a list that is stored in memory (the items are added to a list in memory normally so that they can be displayed to the user and then they can choose to export to CSV or Excel or HTML file from there). But this FileStream method has its drawbacks - for example the results can't be sorted at all because they're being written out to file as soon as they are created so there's no point where each individual item can be compared against all other items, plus I don't like the fact that the user can't see the results in the GUI first and see if they even want to export them.
I can't help thinking there must be a way to make this actually work and have all the data visible in the GUI without it all being in memory at once, by paging the data to/from disk as needed. I know of other programs that must do this, but I don't know of any .NET programs that do this. I know it happens in the background anyway what with the way virtual memory gets paged to/from the Windows page file as required, but that doesn't help avoid the limit of only 4 GB of virtual memory being addressable to 32 bit programs.
I know I could easily use a FileStream to write the data to my own file, writing each entry to disk as it is found by the program, and then just read "small chunks" in at a time - but my problem is how to make that actually work with a GUI. I mean lets say my in-memory list only has half the actual results loaded at the moment, and the user scrolls down my TreeView (or ListView if they're using the alternate view option in the program) or they choose to re sort a column, how do I make that work on ALL of the data when I can't load all of the data into memory at once? How would the scroll bar even work in a vaguely "normal" way at all considering the Tree/ListView won't ever actually have all of the data in it? I'm using WPF for the GUI (just in case someone suggests using some TreeView/ListView events that are specific to WinForms)
I'm wondering if this is something that is too low level for .NET? If not perhaps someone knows if there is actually some way to do this that is going to result in a decent user experience
Oh and inb4 dunfiddlin tells me I don't actually want to do this ;)