How to store incoming data

  • caching (volatile)
  • fifo, priorities (queues)
  • broadcast-style pub/sub (azure sb)
  • NoSQL (Cassandra, HBase, etc.)
  • Log data
Sometimes you might need to write log data for log-analysis later. You can leverage ETW so that your app can publish events that a listener picks up and writes.

When ingesting content, where will you store it? you may first be tempted to write directly to a database, perhaps SQL Server, MongoDB, or Neo4j, as this might be where your content ultimately fits for future processing. However: Sometimes your database-of-choice may be optimized more for reading and searching than it is for the type of content insertion you're planning. This could result in you needing a larger database footprint than initially thought: More CPU and more memory (and maybe more instances) due to the nature of record-by-record insertion.

Last edited Nov 11, 2013 at 11:46 PM by DMakogon, version 5