Wednesday, January 23, 2019

My last 6 months - making StructD worth the time

For those of you who have been paying attention to this blog might be asking, where is Josiah? Well, I've been busy. Not only did I get to go on a little vacation (much needed), but I've also been making StructD (my fork of Redis) better. How? Watch inline below, or if that doesn't work: https://www.youtube.com/watch?v=dr9XepJqm-4




The short version for those who didn't / can't watch is:
  • Background memory snapshot memory use worst-case is 99.8%+ reduced
    • Old: 100 Gig Redis could use 200 Gigs of memory during snapshot, or +100 Gigs
    • New: 100 Gig Redis could use 100.2 Gigs of memory during snapshot worst-case, really +198 Megs, a 99.8% reduction
  • Snapshots load 20-75% faster
  • Snapshots created 25-65% faster
  • Cold replicas start 50-70% faster 
Other things not covered in the video:
  • More Lua environment hardening
  • Better Lua tracebacks
  • Temporarily removed Lua debugger (transactions means that we can do this another way)
  • Improved Lua transaction performance
  • Lua Aliases:
    • SCRIPT LOAD <script> [<alias name> [NX|REPLACE]]
    • SCRIPT ALIAS <sha1> [<alias name> [NX|REPLACE]]
    • SCRIPT UNALIAS <alias name> ...
    • EVALSHA <alias name> <keycount> <keys ...> <argv ...>
    • EVAL "return CALL.alias_name({key1, ...}, {argv1, ...})" ...
  • Lua and other operations are better behaved with respect to one another (more things can happen during Lua scripts)
  • crc16 - 6x faster than Redis, 33% faster than Mark Adler / Matt Standcliff
  • crc64 - 6-8x faster than Redis, 33-66% faster than Mark Adler / Matt Standcliff
  • TXN.MULTI to go with TXN.EVAL / TXN.ESHA, for rolling back data changes on command execution error
Combined with our earlier SSL/TLS additions, Lua transactions, etc., we've got faster and lower-memory snapshot operations, more convenience in your Lua scripts, and performance comparable to or better than open source Redis with just about any SSL/TLS unwrapping daemon.

I've also been shopping around talks surrounding the engineering choices leading to the performance improvements and memory savings currently enjoyed by StructD. From these choices, there are clear incremental steps towards a further 50-80% reduction in snapshot creation / load time, plus a general 50-80% reduction in memory use overall. I will get there, just not tomorrow.

As announced several months ago, I've been working towards starting a hosting company. Still moving in that direction, but until that is quite finished, I've decided the best way to get there in the interim is to open up shop for both unpaid public, and paid private Redis and StructD support.

So, if you'd like to get Redis or StructD support from me, download StructD, get StructD hosting in the future, or just feel like emailing me on a mailing list again, please feel free to drop by StructD.com to learn more.

     StructD 5.0.0 (8d6715c9/1) 64 bit   Mode: standalone   _______
 _______          Build: 20190104.0019   Port: 6380        |   __  \ _
/  _____)     _    http://structd.com    PID: 2670   _     |  |  |  | \_
| /\____\) __| |\_   _ _____   _     _    ______  __| |\_  |  |  |  |:| \_
| \/___   (__   __) | '_____) | |\  | |\ /  ____)(__   __) |  |  |  | |:| \
\_____ \-_ \_| |\_\)| / \___\)| | | | | || /\___\)\_| |\_\)|  |__|  | | |:|
 \____\ | \  | | |  | | /     | | | | | || |/       | | |  |_______/ \| | |
 _____/ | |  | | |  | | |     | \_|_/ | || \|___    | | |   \________/ \| |
(______/ \|  |_| |  |_| |     \_____,_/ |\______)   |_| |     \________/ \|
 \_____\_/    \_\|   \_\|      \_____,\/  \_____\)   \_\|       \________/