Meshnets - cjdns

Exploring Hyperboria

If you pay attention to the "net-neutrality" issue that keeps popping up you may start to feel uncomfortable about how tenuous this whole internet thing is.  Well, I did at least.  When someone asks "who owns the Internet" the answer we hope for is "nobody" or "everyone" but that's not quite right.  The internet is owned by the companies that own the "tubes" or the backbone infrastructure to the internet.

Remember that old "super highway" it's not quite far from the truth.  Imagine if there is a highway that connects us all, but to get to that highway we take the side-streets from our home through our ISP.  Then that highway is really a network of toll-roads.  So your ISP charges you to connect to the "super-tollway" part of that calculation is most of your traffic will be coming from someplace else back to your home (inbound) and you won't be running a service from home that would generate a lot of outbound traffic.  Among the tollways they trade in traffic.  So they hope to keep outbound traffic to a minimum (debts) and balance it with their inbound (credits)

Sounds pretty nice and capitalistic, doesn't it?  Well, some of those companies that own the "tubes" are also in the business of providing content.  Like maybe streaming movies or music, and for as much as gamers love to watch power-bars, watching that damn loading bar is never fun.  So a company might be tempted to prioritize their services over a competitors (one who doesn't own any tubes).  Or, as it happens, they hold data-rates hostage until popular services pay up for all that extra data, you know, the stuff you're already paying for.

Now, laws are good and handy, but nobody really really trusts a legal system to always do the right thing.  I know I don't.  For all the good intentions, there's probably twice as many bad intentions crafting our laws.  So consider instead a system that doesn't require laws to make it fair.    

A small detour

Whatever your opinion of Bitcoin, the purpose was to create a network without trust.  Trust is good isn't it, why don't we want it?  Whenever you go to Target you swipe your card and you trust that they aren't transmitting all your vital information un-encrypted over open networks.  So maybe trust doesn't work out so well after all.  All our traditional banking is built on old systems of trust.  There are trusted middlemen who get a generous cut of the transaction for their trustworthiness.  Bitcoin upgrades this system distributing the trust to the network of users.  Without going into painful detail, basically anyone has the ability to verify a transaction is legit, like a super-power that lets you detect counterfeiting. 

Back to work

So imagine how the Internet could be upgraded.  More importantly what needs fixing?  Security or privacy is a big problem.  There are great application developers out there, but not everyone of them is a good security expert, or believes they need one.  When your favorite social networking grows from obscurity to universal use, it might not have considered security when it was new.  Now that it's big, it has consumed more and more of your private data, but it might be riding on duct-tape security.

Second, getting back to the original problem, what do we do when our Internet service providers decide to play god with the data?  What about less than freedom loving governments trying to inhibit the free flow of information?  If the Internet remains centralized the owners have the power to censor what isn't in their favor.

Imagine if you will, that you live in a small remote town.  One day a cable hungry back-hoe cuts the link connecting the town to that super-tollway Internet.  You're a small town and big internet company decides it's not a priority to reconnect you (though they keep billing unless you go online and submit an interruption of service form, oh wait, no internet...) so your town is completely cut off from modern communications.

Luckily you have a very smart mayor and she decides this is the moment to introduce a town-wide wifi system.  Everyone gathers for a pick-nick and seminar to teach everyone about this new system.  She proposes to team with the local maker/hacker club to help convert the town's home wifi routers into a secure mesh network.  The process upgrades your existing firmware and allows you to turn your wifi router into a mesh-node.  This node will automatically connect to your neighbors in range and create a local-network.  This mesh connection continues street after street until the whole town is now connected again.  It might even work with mobile phones, as you roam the network.

However this isn't the world wide web, it's just a town-wide-mesh.  Local business can re-host their websites on the town-mesh, The mayor might even decide to offer e-mail hosting, your-name@town.mesh.  

This town doesn't exactly exist, but there is a little mesh-community of a few thousand nodes.  We call it Hyperboria, but don't get too excited.  This community is reserved for people ready to make contributions.  It's not a dark-net full of disgusting and illicit activity, in fact it's more like a small town even though it's distributed globally.  For one, it's social, no one has a right to be connected (we call it peered) if you're obnoxious your peers can (and will likely) un-peer you, leaving you disconnected.  There aren't a lot of services on Hyperboria either.  We've got a few of the essentials like github (we're all about development) and IRC.  You can find a social media site and an e-mail service and a few other odds and ends.  Its a slow project, focused on doing it right, rather than right now.

So go back to our town-mesh, once they are re-connected to the Internet, they could keep using their town-mesh while enjoying infinite cat-videos of the web, the two can live side-by-side.  However, one could imagine a newspaper picks up the story and other towns decided it's a right fit, in fact they decided to connect their towns into the same mesh.  Town after town is added to the mesh, close neighbors may establish wireless or fiber "backup" links between towns, these links increase the possible mesh paths for traffic to pass.  In the event of another killer back-hoe, a town in the mesh may not even notice the interruption.  The bigger the mesh with redundant paths means a more robust network for all.

In this little fiction, a geographic area is connected with a redundant series of de-centralized connections.  There isn't a super-tollway, but an actual Web/mesh of connections where data travels via the shortest available path with the power to change that path as the situation changes.  The owners of the internet may still exist but we won't be held hostage to their oligarchy.

Other thoughts

While we're discussing the decentralization of information, it's useful to consider similar situations.  I'm not a fan, but I know the world loves Facebook.  Facebook is a service that allows people to communicate, in exchange for control over everything you'd like to communicate.  If the service decides your choice of communication is not in their interest, they can/will silence you.  Most people, well frankly, aren't interesting enough to fear censorship, however perhaps they live in countries that are inclined to pressure services to censor or relinquish customer data.  Perhaps they choose a lifestyle or use a name other than their legal one, these harmless people can have their digital 'self' confiscated with recourse.  Too bad!

We don't actually need these centralized communication systems, e-mail is still popular and it might be a surprise to most people but you don't need google or aol to send an e-mail, you computer is perfectly suitable to sending and receiving them.
There aren't a lot of distributed substitutions that satisfy every modern social-networking need, but there could easily be, one major limiting factor is the available address space.  There are a finite number of IP addresses in the IPv4 range.  They are controlled and sold, but they are scarce and valuable.  There are alternative like IPv6 which would allow us to address every atom in the solar system with room to spare, but that would make the IPv4 address worthless.  You've likely guessed who's well invested in the IPv4 address space, the same owners of those super-tollway tubes.

Last bits

I'm not a tin foil hat sort of guy, just an engineer who hates to see a poorly designed system hobble on for no better reason than someone is making a boat-load of my money to provide me that faulty service.  There are still a lot of problems to address with a mesh-based internet, consider the incentives for hosting a robust node (one that can handle a lot of traffic) opposed to being a leaf-node (node that handles little-to-none traffic for the mesh) what incentive does one have to bury a fiber line to the next town or create a long range wireless link to the remote local-mesh?  Ideas are floating about some sort of data-rate currency that allows nodes to accumulate points for the data they service, then spend them for priority data on other nodes.  How would you make sure people don't abuse the system?  How would you make sure everyone could get some useful amount of data?  

The system Hyperboria is running on a protocol named cjdns, (cjd are the initials of the inventor, ns is network system/suite)

Development is mostly isolated to *nix systems (cause it's the best :) and Mac, but there are resources for other OSes.  

If you decided to try it out, you'll need to know how to compile and run the service:

https://wiki.projectmeshnet.org/Getting_started#Installing_and_setting_up_Cjdns

I recommend this howto, it will allow you to run one instance of cjdns on your network but all your LAN clients will be able to browse/connenct to the mesh network.

https://github.com/cjdelisle/cjdns/blob/master/doc/nat-gateway.md

Just getting a node running doesn't get you into Hyperboria, in fact you don't actually have to join the community.  Maybe you have a big (nerdy) family, you could create your own private/secure family meshnet providing services only your inner-circle can access. (plus it provides an external IP to your mesh if you don't have an IPv4 already)

If you would like to join Hyperboria, learn about IRC and connect to EFnet/#cjdns or #projectmeshnet.
Important!  Don't go demanding peers, make some friends, ask questions, be neighborly!  If you can manage to make a friend or two, then ask to peer with them (they might just offer).
*when passing peering credentials it's likely your new friend will want to send them encrypted via email, if you don't know how to use stuff like PGP, google it :)

Once your in hyperboria, this clearnet site has a few hype-links to resources inside:
http://hyperboria.be/

Mining to Bits

The Game of Losers

As a hobby bitcoin miner, I find myself wondering if this was how it was supposed to be.  When I began mining I had only a pair of GPU and some simple grasp of the mining game.  I started early 2013 so it was good but not great.  By mid 2013 I had about 0.5 BTC and I was eagerly waiting for my new ASIC hardware to arrive.  My daily mining returns very low, sometimes I didn't even bother to check the miners.  Then I got my ASIC, my hash rates went from 80MH to 6,600 MH, and my returns peaked just a little above my early GPU returns *sigh*, at least I was back in the game.  Time went on, and difficulty went up, I noticed a slump in ASIC prices and I pounced on a Block Erupter Blade and scored myself another 10.7GH.  Now a few months after that, I'm seeing returns less BTC than my electricity use.  Luckily it's only a few bucks and we waste more electricity on our old fridge than on BTC mining.

Was it supposed to be like this?

So I wonder what the mysterious father of bitcoins had in mind.  Did mining pools help or destroy the mining process?  Was the it unthinkable that we would develop single purpose ASICs to feed the mining beast? Right now, if you don't want to drop several thousand dollars on hardware, you're lucky to find $40-50/GH to get about 10GH.  Unless the BTC/$ rate goes way up, this won't pay for electricity for very long.  If you've got more money than sense and can drop $12K or more on hardware, there are some rigs that will get you to $4-5/GH, I've seen as low as $3.   Perhaps I'm just "cheap" but that's more than I want to spend on something that will decrease in value faster than my car.  Plus, I can't re-purpose it for something like gene-folding or alien hunting.  Before anyone considered ASIC bitcoin miners, I think proof-of-work seemed like a great way to link bitcoin with progress in computer power.  ASICs are great tools for specialized work.  Perhaps bitcoins are great for the ASIC industry in general.  On the other hand it sidesteps the "natural" link between the bitcoin network and computer power.  Perhaps that's just romantic thinking.

How would you fix it?

I'm glad you asked.  Frankly, I don't know exactly.  It would be interesting to see a crypto-coin with an adaptable proof-of-work algorithm.  If I'm not over simplifying it, bitcoin uses the sha256(sha256("string")) and Litecoin uses something called "Scrypt" but already there are whispers of Scrypt ASICs as soon as the price of LTC makes it profitable.  If "NewBitcoin" used sha256**X would that break the ASIC market?  If NewBitcoin could flex the hash-algorithm if it senses any drastic change in network hash-rate it might be resistant to specialized, disruptive technologies.

What would changing block rewards instead of difficulty do?  For instance if bitcoin locked difficulty, but divided up rewards, if the average minutes per block went from 10m/block to 1m/block then adjust the block rewards to 10% of the normal block reward.

What is the goal?

I guess before I get too far down the rabbit hole, I need to state what my goal is.  If bitcoin mining gets centralized to a "mafia" of major miners, it defeats the whole democratization of bitcoin.  I don't think there is any reason to distrust big miners, they are likely the most enthusiastic players, but if the whole point is to include everyone, it's not really working.  What is a logical prediction for "commercial" mining?  When the block rewards half again will the commercial miners survive?  Will the ASIC manufactures be able to sell hardware that is half as rewarding?  That calculation depends on the unpredictable variable, what will 1BTC buy next year?  If one BTC twice as much, then the commercial model survives, if not? who knows.
So regardless of the survival of Bitcoin, I'd like a system that rewards many users but discourages major users.  This appears to be a mutually exclusive goal, any system with diminishing returns would be defeated by splitting power over many puppet accounts.  So I'm back to believing the goal of NewBitcoin is to defeat future innovations in mining technology that isn't run on Turing complete hardware.  The network must be flexible/dynamic enough to discourage over-specialization.

Why do you hate ASIC?

I own a pair of Block Erupters I got when they were cheap.  I've got a Jalapeno from BFL before they bumped the price, I got a Block Eruptor Blade when the price was at the bottom (I'm a lucky guy huh?).  Trouble is, all these toys will only ever be used to mine bitcoins (or a sister-coin).  All that power will never hunt for aliens or fold genes.  It won't even help me crack a password, utterly useless hardware.  The kicker is, bitcoin doesn't need ASIC, it doesn't need GPU's or FPGA.  If it was cool to only mine on old laptops, that would still get the job done, no matter how popular bitcoin gets.  It's just an arms race, I recommend investing in the companies that build the miners.

I've also heard that hash power has gotten more efficient, less power is wasted as heat, and we get more hash for the same power.  One block erupter had 4x the hash-power as my GPU and used only 2.5W, so it's obviously better to use the BE than to waste electricity on the GPU.  However, the math doesn't quite work that way, if I've got a power budget of 100W based on GPU mining, I can get ~40 BE's and still make my budget.

That extra hash power doesn't help Bitcoin, it just increases the difficulty next month.

Bits about Bitcoin

Why?

About a year ago I revisited the whole "Bitcoin" thing, the first time I heard of it, I tried to get started but it was technically over my head and new enough the help wasn't helpful enough for my simple mind.  However I learned enough to realized I didn't know jack about cryptography or economics.  After doing some Google-study I eventually learned enough to become more comfortable with the concept.  Unfortunately I began mining with my old GPU in the early part of 2013, just as things were heating up.  I didn't trust any existing exchange with my USD money, yet.  As you likely know, 2013 was a good year for Bitcoin, as well as the dozens of sister-crypto-currencies out there.  I kept my ear to the ground on any articles written on the subject.  At first they were confused, then dismissive, then there was a good flow of Pro and Con.  They all had the obligatory "history of bitcoin" which got pretty tedious after the 100th time.

I'm an engineer/hacker/nerd and as interesting as economics are, I won't pretend to comprehend the models used in the subject.  As a software hacker I prefer to write my code to function with as little user input as possible, people are dirty nasty things that foul up every system. Economists are pretentious enough to presume they can model people, so I remain suspicious of anything "known" to the school of economics.

That said, the "why" I'm want to write this, is I'm a little tired of all the flawed models (straw men) presented in many Anti-Bitcoin articles.  I guess it's understandable, a debate over currency has never been so lively, but I'll try all the same.

Technology

The technology behind Bitcoin is pretty solid.  There are theoretical vulnerabilities, but when compared to common money, it’s pretty much uncounterfeitable.  It’s designed to be anonymous but it’s generally not practiced that way.   The “mining” of bitcoins is a reward system for perpetuating the Bitcoin transaction network.  The miners act as a distributed “bank” validating a transaction just like when you swipe your card at check-out.  It was designed to reward more early and reward less later on when they could be worth more.  Additionally, a optional transaction fee (think tip) for a BTC transaction is incentive to the mining network to process your transaction now.  A safeguard to the network restricts a new block to 1MB which will include a finite number of transactions per block.  

Money-systems

To reiterate, I'm not an economist or a writer, but as an engineer, economies look like systems where goods/services flow in one direction and money flows in the other.  There are good money-systems and there are bad ones.  If you don't see the point of Bitcoin then you likely regard your money-system as a good one.  Likewise if you think "hey I should get my money out of the bank and get some BTC" then you're probably in a bad money-system.

If you live/work in a good money-system, you’ll buy and pay taxes with the same money you are paid.  Within that system, you won’t notice the relative volatility because everyone is gaining and losing absolute value at the same rate.  Unless you trade with another country or money, then you’ll notice the volatility.  These effects will slowly change the price within your local money-system like higher/lower prices on imports.  If you have bad money-system, your money will loose value so fast you’ll notice immediate effects within your money-system.

  

In a “good” money-system, there is little need/reason to adopt crypto-currency, in fact a hybrid money-system would cause the normally insulated money-user to feel the relative volatility between currencies.

Philosophy

Where things get spirited is the philosophy.  The BTC and the USD run in bipolar economic models.  As an engineer I see the USD working like this.  Greatly simplified:

1.       Source: The USG prints USD and spends them

2.       System: USD circulate in the US economy

3.       Sink: Taxes remove USD from circulation.

This might look weird; because taxes don’t pay for public services, taxes simply restrain inflation.  The USG is effectively outside the money-system, while we lowly people are stuck keeping to a budget.

This money-system is designed to devalue over time, just enough to motivate the individual to consume now as each USD will be worth less later.  According to Keynes, an economy works better when people consume more now instead of saving for later.  Most economists, banks and governments agree with Keynes.

Bitcoin (and many, but not all of the other cryptocurrencies) are engineered in a different way.  First there is a cap on the number of BTC.   Second the production is prescribed by mining.   Another overlooked fact, there is a network cap of 21 MBTC, but for every lost wallet, those BTC are also lost.  I suspect that many of the early miners generated hundreds or thousands of BTC each and eventually deleted the wallets when they moved on to other hobbies, one poor guy threw is hard drive out.  Fortunately, as an infinitely divisible money-system, the whole global economy could fit under any fraction of the total BTC pool.  

If the USD was run like BTC, things would be different, instead of a source-system-sink model; the government would be on equal ground with any other user in the money-system.  Meaning to spend money on public services, the USG would need to collect taxes or other revenue; you have to have BTC to spend BTC.  It is obvious why a government wouldn't care for this burden, so I doubt an entrenched political class will make a move to Bitcoin.

Straw-men

A pet peeve of mine is listening to persuasive arguments that make heavy use of logical fallacies.  If you listen, they saturate our media, so much that I'm sure I've learned to be as just as guilty in habit, but I try to be better.

One particular problem I see recurring is attacks against BTC/USD volatility.  One annoying example was a theoretical BTC-mortgage.  You buy a house with a BTC mortgage, when 1BTC=1USD.  Then the value jumps to 1BTC=100USD.  Your mortgage just jumped two orders of magnitude!  This argument assumes a mixed-money system where you, the home-owner, are paid in USD but then convert that to BTC to pay your mortgage.  In a good money-system there is no reason to pay your bills using BTC if your wage is USD.  If you are in a bad money-system, you might stash your wage in BTC to save it's value, but a mortgage in a non-native currency would be risky.  If you are in a Bitcoin money-system, your wage in BTC would pay your mortgage in BTC just like it can in USD.  No problem. 

What's it Worth?

Everyone is struggling with this one, but I think I know the answer, and with all answers in engineering, it depends.  It depends on how big the BTC money-system is, not the commodity price.  

If the only country using Bitcoin was the country Galt, and Galt  was as productive as California ($2T in 2012) then the value would be 1BTC = $95,238.  Basically, as more groups adopt one finite resource the unit-value will go up, just as if Canada decided to switch to USD, we’d either have to print more money (increase the supply) or the demand/value would jump by 12% .  The problem with stating this way is it betrays a value-bias to the USD, but since it is a familiar valuable object so I think it works. 

Confessions

I don't know why it matters, but I figure someone may care.  I still have not traded USD for BTC.  I don't like to think of Bitcoin as a commodity.  I chose to invest lightly in ASIC miners.  The hardware business of Bitcoin is a whole other monster, perhaps another post.  I bought my first ASIC from BFL, I fretted over BFL using my money to build my ASIC and then mine BTC with it while difficulty was still low and I waited for my product.  It did eventually come.  Later in a post $266/BTC to $70 slump the price of ASICs went down and I grabbed a few more before the meteoric rise to $1200/BTC.  Now the $/GH has created a chasm between hobby miners like me and serious investors.  If you have the resources you can buy $4-5/GH.  If you want more modest hardware you will pay $40-50/GH.  

The Merits of Mineral Oil

A work in progress

I've been experimenting with oil-cooled computers.
It started with a basin and an unloved mother board, that worked out just fine.

Eventually I bought the $30 fish-tank and sneaked a bottle or two of mineral oil in with the groceries (the wife didn't officially sanction the project)

Dunking isn't a new thing (although many haven't heard of it) but most of the projects are done for the shock or the possibility of limitless cooling.

[quick note: mineral oil is totally safe on the solid-state components, don't mix moving parts or hard-drives.  The disks may appear to be sealed but they will have a small filtered port that equalizes the pressure in the disk, oil will get in that way]

[another note water is pretty safe on your solid-state components just so long as power is not applied, if you change your mind about oil-dipped hardware, it will take some time but you can wash it off, just let it dry for a month or more *I'm not responsible for your stuff.]

I too thought it was shocking, but I wanted to know how it worked.  Looking into the thermodynamics I learned a few things.

1. Oil conducts heat well, better than air, not as good as water (or aluminum)
2. Oil requires slightly more heat to increase the temperature by 1 deg than air per unit-mass
3. Oil has much more mass per unit-volume .: holds much more heat per unit volume than air (still less than water)

A bit of envelope math showed that my 5-gallon fish-tank had the thermal-capacity of a bedroom or small office (10x10x8) or so.  The downside is the office/bedroom will exchange that thermal mass more easily than the oil.  The upside is all that waste heat is contained, and controllable.   This means by containing and controlling the fluid I control the heat as well.  Now I can decide/engineer a way to dump the heat.  I don't like fan noise, perhaps I use a large passive radiator then pump the oil through that.

The Motivation

I decided to write this journal because most of the information I found was fragmented and littered with inaccurate advice, or people assuming different motivations.  For myself, I wanted a modestly powerful computer that ran silent.  I've never bothered to overclock a PC, I don't spend top dollar to buy hardware, if I really needed that performance I'd spend a little more.

I'm also interested in the server application.  We've got some sever rooms where I work and they are horrible places to visit, I would hate to work in one.  There were some tests with oil-cooled servers that proved to be more efficient at controlling temperature.  Not only that but the rooms were quiet and normal room temperature (not freezing and windy).

It's not magic, the heat/energy must go someplace, but with a submerged system, where it goes is quietly controlled.  There were some other benefits like dustless and low-vibration that would likely mean hardware lives longer.

That's a bit down the road, but for now I'm curious how a computer responds to load and heat.

The Fan

One bump in the road I hadn't expected, the mini-power supply had a fail-safe in case the PSU fan was to fail.  I discovered this when the oil covered the PSU slowing the fan (increasing the current) and causing the PSU to trip and shut down.  The trick was first measuring the normal current of the fan, that was about 0.3 mA (if I remember right).  Then I put my finger on the hub slowing it down, the current rose to 0.4 mA and did not trip, but as I slowed it more and hit 0.5 mA the PSU tripped.  So I calculated the resistor value to simulate a 0.25 mA "fan" and replaced the real part with the simulated one.

The Tests

I ran a cron that uselessly calculates primes for 30 minutes of each hour (heavy CPU) then rests for 30 minutes.  Looking at the plots it looked like the system was getting warmer over time but not catastrophically.
I still monitor and plot the temperatures just for giggles.  You can see that it's running around 40 C and can get a bit warmer.  I run this cron on some of my air-cooled systems to get an idea of how they respond.

Normal monthly load (not the stress test)

Monitoring the systems I'm learning that each machine has a different thermal signature.
For instance the net-top does not get much hotter under load, but it runs pretty hot at idle.
The gaming machine runs much cooler all around, but fluctuates rapidly responding to small changes in  CPU load.

The Oil-PC (an old shuttle stripped down to the frame) has a much more stable profile.
The system can only release heat from the walls of the fish-tank (less than optimal).
When the Oil-PC comes under load it will jump and level off at the high-temp.  Going back to idle the CPU cools quickly to idle-temp and holds there with little fluctuation.

The Ooze

So I had a "great" idea to mix phosphorescent powder with the mineral oil.  I got about 5g of blue/green powder.  The mixture tested to be non-conductive, and it glowed nicely but it wouldn't stay in suspension.  The result was glow-in-the-dark "dust" at the bottom of the tank.  So I'm not going to do that again.  I had hoped that I could get the oil to "glow" from within but even when the powder was still suspended it just looked like dusty specs floating in oil.  Not the "secret of the ooze" I was hoping for.

What's Next?

I'm looking at methods to build a custom tank for my gaming system.  I'd like have a bit better presentation and cable management .

Oops!

I finally got around to connecting up a pair of radiators to a 50gph fish-tank pump.  I picked the pump mostly for dimensions and hoped for the best.  I bought some surgical tubing (10') and connected everything up.  It was less than impressive.  The flow rate was there but unimpressive.  I left it running and did not observe any significant changes.  After a few days though I noticed the hose detached itself from the pump.  I went to re-attach it and noticed it was now entirely too large to fit snugly over the diameter.

I figured the warm oil bath caused the rubber to expand, luckily the pump came with a larger diameter attachment and it fit again.  Until I noticed the oil now had an odd yellowish tint.  I worried the heat was dissolving part of my computer, until it dawned upon me... Surgical tubing is the same sort of sickly yellow color.  So I Googled surgical tubing and mineral oil and I was reminded that mineral oil has a nasty habit of damaging latex condoms!  Egads!

So note to self, latex and mineral oil don't mix, or rather they mix all too well.

The Myths

So I want to keep a running summary of the myths I read on posts and what I've experienced with them

1. It will damage hardware:
• No.
• Also no fish, don't be silly.
2. It's too messy to deal with
• If you are not prepared it will be messy.  It's not so bad getting your hands messy, quite good for the skin!
• If this becomes a big thing for me I suspect I'll build a hanger that I can suspend the hardware while I replace parts allowing the oil to drip back into the tank without making a mess.
• So yes, it's a bit messier, but I wouldn't recommend it for a laptop, but perhaps a server that you keep in the dungeon.
3. It's not as good as water-cooled:
• Yes the thermal conductivity isn't as good.  If you aren't overclocking or worried about catastrophic temperatures then it's quite good.  
• Oil is better than water for submersion!
• It really depends on how you engineer the system.  My next rig will test means to transfer heat out of the system.
4. Post-submersion connections:
• I have had no trouble connecting video-cards, USB, PS/2 or any connectors after dipping or while submerged.  It does take some care to remove oil from my hands but otherwise it's fine.
5. Upgrades are impossible
• Messy yes, impossible no.
• If you are lazy like me, you could just dip your arm into the tank and swap out the ram or a video card, it will make a connection, just let your arm drip and have a towel ready.
  

Oil Drilling... er mining

So I replaced the tubes with nice clear vinyl tubing which was useful for locating and eliminating bubbles.  The bubbles acted like blocks in the oil flow, a stronger pump would have handled things better.
With the radiators and pump I wasn't able to "over heat" the system with any amount of CPU use.  I decided to mine litecoins to heat-stress the oil, they don't quite make money, but it feels more productive than tossing primes into /dev/null.

For a bit I added a GPU (too old to mine bitcoins) and it added tons of heat.  It was an old card that had the fan burn out from dust.  So now the bare chip radiates heat into the oil.  In fact you can see the distortion in the column of hot oil streaming off the chip.

I picked up a block eruptor (small bitcoin mining ASIC)  for cheap.  Its a little USB device, 333MHash/s but it gets pretty hot.  Before I dunked it I tested it by attaching a CPU heatsink to the back.  It was awkward but kept the device from getting too hot to handle.

So obviously I had to dunk the mining ASIC.  I decided to remove the useless and power-hungry GPU and cut off the CPU-mining.  Immediately the temperature drops.  I don't have a way to measure the exact temperature of the ASIC but I can guess it's near enough the average temperature of the mobo sensors I use to plot my temperatures.  The ASIC does report a temperature but I'll have to see how accurate it is.

Update 2013-12-31:

I got a IR thermometer for Christmas, so I can now get some measurements!

• Tank-wall = 43C
• Oil = 50C
• Ambient =16C
• Air-cooled BE (333MH) = 80C (hot!)

I'm now running a Block Erupter Blade (10GH) in the oil bath.  I've moved the smaller Block Erupters to an air-cooled position on another server.  Last I've got one BFL Jalapeno in a standard air-cooled position (not enough room to dunk it!)

Minor Issue

I have noticed that some of the PVC insulated cables (like on a keyboard) that are submerged got stiff and brittle after some time in the bath.  Not an issue while soaking but they failed pretty easily with normal use.  I'm not certain what the insulation is made of, but PVC is supposed to be inert with mineral oil.  I'll update if I ever figure it out.