Hi, I'm creating a console application and I'm looking for the best way to upload files in chunks to a server with multiple connections (10 connections maximum). The files are uploaded as strings and not bytes. All files in a folder need to be read/encoded in a separate thread and added to a thread-safe queue. Each upload connection grabs a chunk from the queue and uploads it to the server, etc, etc, until all files have been uploaded ==> "Producer/Consumer". I don't want to read all data at once, because the files might be a few hundred megabytes.
Steps.
Before spending more time on this I'd like to know what the best way is to do this, because I have a feeling that there's a better way.
Also, what if the uploading goes faster than the file reading/encoding? What is the proper way of letting the upload connections wait until there are items in the queue again?
Steps.
Code:
1) Create a separate thread to read chunks from all files in a folder.
2) Encode the chunks to string (custom encoder) and add them to a queue.
3) Temporarily halt this process when a certain number of items are in the queue.
4) Establish several multi-threaded connections to the server.
5) Each connection grabs an item from the queue and uploads it.
6) Continue reading/encoding chunks until a certain number of items are in the queue again.
7) When a chunk has been uploaded, then grab the next one and upload it.
8) Repeat steps 6 and 7 until all files have been uploaded.
Also, what if the uploading goes faster than the file reading/encoding? What is the proper way of letting the upload connections wait until there are items in the queue again?
vb.net Code:
Imports System.Threading Imports System.Collections.Concurrent Imports System.IO Module Program Public _queue As New ConcurrentQueue(Of String) Public _waitHandle As New AutoResetEvent(False) Sub Main() ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf ReadEncodeFiles), "C:\Folder") For i As Integer = 1 To 5 Dim c As New Connections("server.com", 1500) Next 'Wait here. End Sub Private Sub ReadEncodeFiles(ByVal state As Object) Dim folderPath = DirectCast(state, String) Const BUFFER_LENGTH As Integer = 500 * 1024 Dim buffer(BUFFER_LENGTH - 1) As Byte For Each filePath In Directory.GetFiles(folderPath) Do Until byteCount = 0 'Encode chunk to a string with custom encoder. Dim encodedData As String = EncodeToString(buffer.Take(byteCount).ToArray()) 'Add encoded chunk to the queue. _queue.Enqueue(encodedData) 'Wait if there are 20 or more items in the queue. If (_queue.Count >= 20) Then _waitHandle.WaitOne() End If Loop End Using End If Next End Sub End Module
vb.net Code:
Imports System.Net.Sockets Imports System.IO Public Class Connections Private _socket As TcpClient Private _stream As Stream Private _databuffer(1023) As Byte Public Sub New(ByVal server As String, _ ByVal port As Integer) _socket = New TcpClient _socket.SendTimeout = (30 * 1000) _socket.ReceiveTimeout = (30 * 1000) _socket.BeginConnect(server, port, AddressOf ServerConnect, Nothing) End Sub Private Sub ServerConnect(ByVal ar As IAsyncResult) Try _socket.EndConnect(ar) _stream = _socket.GetStream() _stream.BeginRead(_databuffer, 0, 1024, AddressOf Read, _databuffer) Catch ex As SocketException End Try End Sub Private Sub Read(ByVal ar As IAsyncResult) Try If _stream IsNot Nothing Then Dim byteCount = _stream.EndRead(ar) If byteCount = 0 Then 'Server closed the connection. Else 'Communication with server not added yet. 'Upload the next chunk of data from the queue. Dim Chunk As String = String.Empty If (_queue.TryDequeue(Chunk)) Then 'Send next chunk of data. Send(Chunk) 'Signal 'ReadEncodeFiles' to continue reading/encoding chunks of data. _waitHandle.Set() Else 'Queue empty ?? End If _stream.BeginRead(_databuffer, 0, 1024, AddressOf Read, _databuffer) End If End If Catch ex As Exception End Try End Sub Private Sub Send(ByVal message As String) Dim buffer As Byte() = System.Text.Encoding.GetEncoding("iso-8859-1").GetBytes(message) '1252 Try _stream.BeginWrite(buffer, 0, buffer.Length, AddressOf Write, Nothing) Catch ioex As IOException End Try End Sub Private Sub Write(ByVal ar As IAsyncResult) _stream.EndWrite(ar) End Sub End Class