I'm having a strange issue with multithreading in VB.net. It'll take a bit of explanation, so bear with me.
I'm writing a VB application that receives JSON messages pulled from a remote server. The throughput is extremely high, on the order of hundreds of messages per minute, and it is vital I store them all in a database. Obviously, storing them directly after receiving them would take far too long, so I've split up the tasks between threads.
The 'stream' thread receives the messages, decompresses them, and passes the results to a 'message thread' in a thread pool. Each message gets it's own thread in the pool. The 'message thread' then performs some validation and stores the results in a concurrent dictionary called activeOrders, and another copy in another concurrent dictionary called ordersPending. Before the program begins, activeOrders is populated with whatever orders are currently in the database. Meanwhile, a 'database updater' thread pulls it's own local copy of ordersPending and clears the dictionary, like this:
The 'database updater' thread then goes through marketData and adds it to the database. 'Database updater' is on a forever loop, so once it completes the updates, it grabs whatever has been put in ordersPending in the meantime and starts again.
The idea is to have 'database updater' only concern itself with the orders in ordersPending, rather than loop through the entire activeOrders dictionary, as not every entry in activeOrders will need to be updated.
The issue I'm having is that after a few minutes, activeOrders has over 10000 elements, while 'database updater' has only placed 10 or 20 into the database.
Here's the database updater thread:
Here's the message thread:
DBUpdater is completing very frequently, so it's not getting hung up anywhere, and when I put a breakpoint at the statement that grabs a local copy of ordersPending, it only ever has a few elements. I realize this is a lot of code, but I'm not sure if my problem lies in misconceptions about how the threading works, or if it's simply a bug. If any further information is needed about how the code works above, I'll try to explain. In case it matters somehow, orderIdList is populated with all the current orderIDs in the database, and is updated as DBUpdater adds and deletes from the database.
I'm writing a VB application that receives JSON messages pulled from a remote server. The throughput is extremely high, on the order of hundreds of messages per minute, and it is vital I store them all in a database. Obviously, storing them directly after receiving them would take far too long, so I've split up the tasks between threads.
The 'stream' thread receives the messages, decompresses them, and passes the results to a 'message thread' in a thread pool. Each message gets it's own thread in the pool. The 'message thread' then performs some validation and stores the results in a concurrent dictionary called activeOrders, and another copy in another concurrent dictionary called ordersPending. Before the program begins, activeOrders is populated with whatever orders are currently in the database. Meanwhile, a 'database updater' thread pulls it's own local copy of ordersPending and clears the dictionary, like this:
Code:
Dim marketData = ordersPending
ordersPending.Clear()The idea is to have 'database updater' only concern itself with the orders in ordersPending, rather than loop through the entire activeOrders dictionary, as not every entry in activeOrders will need to be updated.
The issue I'm having is that after a few minutes, activeOrders has over 10000 elements, while 'database updater' has only placed 10 or 20 into the database.
Here's the database updater thread:
Code:
Public Sub DBUpdater(ByVal tempList As Object)
Dim orderIdList = CType(tempList, List(Of Long))
While True
Console.WriteLine("DB Update started...")
Dim transactionData = transactionsPending
For Each item In transactionData
Dim temp = CType(item, transactionData)
transactionsPending.TryTake(temp)
Next
Dim deleteOrders = deletedOrdersPending
For Each item In deleteOrders
Dim temp = CType(item, Long)
deletedOrdersPending.TryTake(temp)
Next
Dim marketData = ordersPending
ordersPending.Clear()
Dim size = marketData.Count
Dim count = 1
For Each row In marketData
If (orderIdList.Contains(row.Value.orderId)) Then
nonQueryDB("nema", "UPDATE activeorders SET price=" + CStr(row.Value.price) + ", volLeft=" + CStr(row.Value.volRemaining) + ", uploadDate='" + CStr(row.Value.generatedAt) + "', updateCount=updateCount+1 WHERE orderId=" + CStr(row.Value.orderId))
Else
nonQueryDB("nema", "INSERT INTO activeorders (orderId, itemId, price, volLeft, volStart, minVolume, `range`, buy, duration, issueDate, uploadDate, stationId, systemId, regionId, updateCount) " &
"VALUES (" & CStr(row.Value.orderId) & ", " & CStr(row.Value.typeId) & ", " & CStr(row.Value.price) & ", " & CStr(row.Value.volRemaining) & ", " & CStr(row.Value.volEntered) & ", " & CStr(row.Value.minVolume) &
", " & CStr(row.Value.range) & ", " & CStr(row.Value.bid) & ", " & CStr(row.Value.duration) & ", '" & CStr(row.Value.issueDate) & "', '" & CStr(row.Value.generatedAt) & "', " & CStr(row.Value.stationId) &
", " & CStr(row.Value.solarSystemId) & ", " & CStr(row.Value.regionId) & ", " & CStr(row.Value.updateCount) & ")")
orderIdList.Add(row.Value.orderId)
End If
'Console.WriteLine("Completed update/insert " & count & " of " & size)
count = count + 1
Next
size = deletedOrdersPending.Count
count = 1
For Each row In deleteOrders
If (orderIdList.Contains(row)) Then
nonQueryDB("nema", "DELETE FROM activeorders WHERE orderId = " & CStr(row))
orderIdList.Remove(row)
'Console.WriteLine("Completed deleteing " & count & " of " & size)
End If
Next
size = transactionData.Count
For Each row In transactionData
nonQueryDB("nema", "INSERT INTO transactiondata (stationId, systemId, regionId, itemId, volume, price, date, buy) VALUES (" & CStr(row.stationId) & ", " & CStr(row.systemId) & ", " &
CStr(row.regionId) & ", " & CStr(row.itemId) & ", " & CStr(row.volume) & ", " & CStr(row.price) & ", " & CStr(row.transactionDate) & ", " & CStr(row.buy) & ")")
Next
Console.WriteLine("DB Update complete")
End While
End SubCode:
Public Sub updateOrdersThread(ByVal rowSets As Object)
Dim temp = CType(rowSets, ArrayList)
For Each rowSet As Dictionary(Of String, Object) In temp
'Record temporary message data
Dim tempTypeID As Integer = CInt(rowSet.Item("typeID"))
Dim tempRegionID As Integer = CInt(rowSet.Item("regionID"))
Dim tempGeneratedAt As Integer = convertToTimestamp(CStr(rowSet.Item("generatedAt")))
'If the item is not in base price database, ignore it
Dim basePrice As Decimal
If (priceHistoryCache.ContainsKey(tempTypeID)) Then
basePrice = priceHistoryCache(tempTypeID)
Else
Continue For
End If
'Access item data
Dim row As ArrayList = CType(rowSet.Item("rows"), ArrayList)
Dim newOrderNumbers As New List(Of Int64)
Dim DBorders = getOrderIdList(tempRegionID, tempTypeID)
For Each order As ArrayList In row
'Ignore orders with suspect prices
'Redacted
'Add order ID to list
newOrderNumbers.Add(Int64.Parse((order.Item(3).ToString())))
Next
If (DBorders.Count <> 0) Then
If (DBorders.Count > 10) Then
For Each orderNumber In DBorders
If (Not newOrderNumbers.Contains(orderNumber.Key)) Then
activeOrderCache.TryRemove(orderNumber.Key, activeOrderCache(orderNumber.Key))
deletedOrdersPending.Add(orderNumber.Key)
'Console.Write("Order deleted - DB Order Count: " & DBorders.Count() & " New Order Count:" & newOrderNumbers.Count() & vbCrLf)
End If
Next
End If
End If
For Each item As ArrayList In row
'Ignore flagged orders
If (Int64.Parse((item.Item(3).ToString())) = 0) Then
Continue For
End If
'Populate new marketData object with data
Dim newMarketData As New marketData()
With newMarketData
.typeId = tempTypeID
.regionId = tempRegionID
.generatedAt = tempGeneratedAt
.price = Convert.ToDecimal(item.Item(0))
.volRemaining = Integer.Parse(item.Item(1).ToString())
.range = Integer.Parse(item.Item(2).ToString())
.orderId = Int64.Parse(item.Item(3).ToString())
.volEntered = Integer.Parse(item.Item(4).ToString())
.minVolume = Integer.Parse(item.Item(5).ToString())
.bid = CType(item.Item(6), Boolean)
.issueDate = convertToTimestamp(CStr((item.Item(7))))
.duration = Integer.Parse(item.Item(8).ToString())
.stationId = Integer.Parse(item.Item(9).ToString())
.solarSystemId = Integer.Parse(item.Item(10).ToString())
End With
'Ignore orders with invalid system or station IDs
If (Not systemDataCache.Contains(newMarketData.solarSystemId)) Then
Continue For
End If
If (Not stationDataCache.Contains(newMarketData.stationId)) Then
Continue For
End If
' Update active orders
If (activeOrderCache.ContainsKey(newMarketData.orderId)) Then
If (activeOrderCache(newMarketData.orderId).volRemaining > newMarketData.volRemaining) Then
Dim transData As New transactionData
transData.stationId = newMarketData.stationId
transData.systemId = newMarketData.solarSystemId
transData.regionId = newMarketData.regionId
transData.itemId = newMarketData.typeId
transData.buy = newMarketData.bid
transData.price = newMarketData.price
transData.transactionDate = newMarketData.generatedAt
transData.volume = activeOrderCache(newMarketData.orderId).volRemaining - newMarketData.volRemaining
transactionDataCache.Add(transData)
transactionsPending.Add(transData)
'Console.WriteLine("New transaction data from updated order")
End If
ordersPending.Item(newMarketData.orderId) = newMarketData
activeOrderCache.Item(newMarketData.orderId) = newMarketData
Else
' Add new order
If (newMarketData.volEntered > newMarketData.volRemaining) Then
Dim transData As New transactionData
transData.stationId = newMarketData.stationId
transData.systemId = newMarketData.solarSystemId
transData.regionId = newMarketData.regionId
transData.itemId = newMarketData.typeId
transData.buy = newMarketData.bid
transData.price = newMarketData.price
transData.transactionDate = newMarketData.generatedAt
transData.volume = newMarketData.volEntered - newMarketData.volRemaining
transactionDataCache.Add(transData)
transactionsPending.Add(transData)
'Console.WriteLine("New transaction data from new order")
End If
newMarketData.updateCount = 0
ordersPending.Item(newMarketData.orderId) = newMarketData
activeOrderCache.Item(newMarketData.orderId) = newMarketData
End If
Next
Next
End Sub