AI Data Center Contamination by Larry Good

This is going to be a two-part talk. First I’m going to address “AI Datacenter Water Pollution” and because that will already be long enough, I’ll do a second talk on water consumption. I may have to do a third on electricity consumption and the planning processes for “Large Load” consumers and their effect on grid infrastructure. That’s another area where I am not the definitive expert, but I’ve picked up more insight than the average person, merely by being in the industry and in a risk management capacity, looking at future impacts and technical planning needs.

For background: I work in Information Technology, and most of my career has now been in large infrastructure. Internet banking, large public-school districts, telecommunications, DoD support (including 5 years forward deployed to combat zones) and for the last 13 years, supporting the bulk electric system (‘the grid’) at a large utility.

I don’t work in AI datacenters specifically, but I’ve been doing data center operations, planning, engineering, and cybersecurity support for quite a few years. Running this big chunk of the national bulk electric system grid literally cannot be done effectively and efficiently without computers. From the physical plant perspective, it doesn’t matter what purpose the computing resources are being used for. I’ve done a lot of capacity planning for physical space, HVAC, and continuous & backup power requirements for IT installations. I’m not a civil engineer, but I’ve had to interface and collaborate with many over the years, and for budgeting and resourcing purposes I have picked up a general knowledge of the necessary infrastructure considerations as it pertains to large, multi-year, multi-million-dollar projects.

Currently, the public zeitgeist is all atwitter with panic over the water supply being sucked up into the ‘AI Datacenter infrastructure that will enslave us for 1000 years, and poison the earth forever’. Look, I’m not a huge fan of AI but let’s not be luddites, either.

And as part of the “drank from the garden hose” generation, I have a lot of disdain for the hype about water pollution in this context. It’s (mostly) bullshit.

Singling out this ONE type of commercial/industrial application simply isn’t rational. A large part of it is (as so many things on the Internet currently are, especially with AI making it easier and faster) nation-state and other organized propaganda farms generating FUD for their own purposes. Like those other nation states “winning the AI race”. That’s my cyber-security/critical infrastructure protection specialist hat I’m talking through there.

Yes, data centers in general use more cooling *on the data center floor* than the average office building (per floor, at least. Many office complexes are WAY bigger than most datacenter facilities), but less than the average industrial plant. We have a more concentrated electric usage on floors with server racks (than cubicles), but we don’t have nearly the personnel usage of environmental controls, water fountains, flushing toilets, and inefficient heating/cooling/lighting needs for human comfort and large open spaces. This difference in water usage is significant. Office cubicle farms really require a lot of juice too, with all the individual machines, monitors, peripherals, lights etc. Human density on the data center floor is tiny compared to cubicle farms in office buildings. Traffic is low, with all kinds of knock-on effects like we don’t need janitors vacuuming carpeted floors daily. We don’t need lights on all the time.

Yes, the water flowing through the cooling system is technically industrial wastewater. As one supremely over-confident person told me: “once it leaves the distribution system, it isn’t drinking water anymore.” True: water that has been used in a cooling system is *deemed* non-potable by standard building codes and conservative safety practice, not by tested mineral content.

It is non-potable water. Which does not mean you would be poisoned by drinking it directly from an effluent pipe. It’s a definitional thing. If your kid is homeschooled at the age when your neighbors’ kid is in kindergarten, the neighbor kid is a kindergartner and yours is not. By definition.

Water in your toilet tank (not talking about the bowl- not yet) is the same water coming out of your kitchen tap. Is that water in the toilet tank drinkable? Would you go scoop a glass of water out to drink? Would you drink it confidently in an emergency? (I would). Water coming out of your kitchen tap into your kitchen sink; would you scoop a glass of water out of your filled sink and drink it? What about using your bathtub for storing water in advance of a predicted storm or natural disaster? Would you drink that? I think for most people it depends on need, and they can acknowledge that the reaction of disgust is not rational, it’s emotional and could be overcome by a risk-based evaluation of cost/benefit.

What about water in a freshly-cleaned and sanitized toilet BOWL, that hasn’t been excreted into. Is that water “*polluted*? I wouldn’t want to drink it. But I would have no problem putting it on my garden.

All of these scenarios provide a risk/reward calculation that a person has to figure for themselves, and all deal with less than perfect water quality, but none of them are absolute, and none of this water is “polluted”. But here’s another thing- your municipal tap water isn’t perfect either. You just accept that it is “good enough” and never think about it unless it hits the headlines with a “boil water” advisory. We don’t even know what the perfect mix and amount of minerals in water truly is, or even that there is a perfect mix for all populations under all conditions. Although I’m sure the makers of Brawndo would tell you their sportsball drink is the perfect mix: because it has electrolytes (sodium chloride, iron, magnesium, calcium, etc.).

But yes, industrial/commercial buildings generate water that is not recommended for drinking directly.

Water in cooling towers evaporates. It doesn’t get treated by industrial process. It just rains down. The remnant may have some higher levels of metal & minerals in it than the treated inputs from the municipal supply. So does the water going INTO the municipal supply before it is treated. The municipal supply typically ADDS some other chemicals- chlorine, fluorine. By code, the remnant water from g evaporative towers can be discharged into normal sanitary drains, even if the mineral content is slightly more concentrated. That means decades of engineering and testing has deemed it safe. It is by any rational standard, MUCH less dangerous than your toilet water which gets discharged into the same sort of drain.

Most of the time, the municipal water supply is used by default because it’s a convenient source of relatively clean water to start with. It may even be REQUIRED by local ordinance and city/county planners. Sometimes it still has to be filtered again at the industrial site prior to use. Some industrial processes use de-ionized water. It can’t even have the normal mineral content that humans NEED in our water (trace minerals) and cannot tolerate the additives like chlorine and fluoride used for sterilization and tooth treatment.

Depending on the municipal supply’s mineral content (varies wildly across the USA, as long as it is within EPA tolerances), cost, and delivery availability, a closed loop system might find they don’t want the municipality involved at all. They might have water trucked in, or tap a natural supply from roof run-off or natural water source, and filter it themselves. In that case, the water they discharge could potentially be “cleaner” than what they take in.

In no case are they going to get permitting to release contaminated water that is a risk to the public. And EVERY building must go through a building permit process, and EVERY commercial building gets even more scrutiny for water usage, runoff, even plans for stormwater management.  Alexandria Ocasio Cortez was in the news lately: her jars of brown water stunt were supposedly taken from a site where a datacenter was *under construction*, not *in operation*.

“Rep. Alexandria Ocasio-Cortez displayed two bottles of dirty water that she claimed came from Morgan County, GA, where a Meta data center is being built.”

(Source: https://www.yahoo.com/…/aoc-pulls-two-jars-brown…)

“Being Built.”

AOC didn’t have *data center effluent*. If what she had even came from a municipal water supply (doubtful), she had evidence of local government corruption and incompetence (common), not capitalist exploitation.

Some cooling systems might (not definitely, but may) use piping that may not be certified for drinking water. Perhaps it has soldered connections that aren’t rated for consumption. But closed loop systems aren’t just discharging it. And glycol is expensive. You don’t discharge “cooling chemicals” and refrigerant, you KEEP IT in closed loop systems. Data centers are hardly the only, or even primary user of this tech. EVERY OFFICE BUILDING constructed in the last century, every movie theater, every grocery store, feed store, department store and government building uses it too.

‘Contaminants’ are not necessarily persistent poisonous toxins, but may be anything not necessary for human health in drinking water. Could be irrelevant to human health. Or too much of something you do need: too much salt, too much calcium or iron, and BTW there are actually allowable levels even for arsenic coming into your drinking water from primary sources like aquifers. Parts of the USA have always had arsenic levels.  Right now, there is a democrat effort underway to block testing for birth control hormones and abortifacients in municipal water supplies, because they say “it’s a witch hunt to demonize and ban birth control, and couldn’t possibly be at a level that is harmful.” (Weaponizing Water: How the Campaign Against Medication Abortion Co-opts Environmental Policy | Guttmacher Institute)

Industrial standards already exist for what level of contaminant are allowed to be discharged back into the municipal sewers, and the municipal sewage system then treats before discharging into natural bodies of water or returning to the drinking water distribution system. When it exceeds these levels, it must be treated on site first. These standards even extend to *water temperature* before being discharged into natural bodies of water- but such discharge is almost never allowed in the permitting process anyway.

In the cases where water is contaminated at an industrial building, it simply gets piped across to the building where they process it, and piped back into the cooling loop. Doubly so for silicon fabs. Most industrial processes and any chemical plants do this all day, every day, with far more dangerous “pollutants” than will ever leach out of a modern air conditioner’s guts.

This is a solved problem.

Larry Good, Author at Ordinary Times
Facebook: Lawrence Good, Scribbler

54 thoughts on “AI Data Center Contamination by Larry Good

  1. Past infrastructure improvements, from roads and the development of our air (and before that, rail) systems, to rural electrification and phone system have all faced opposition from coteries of kooks. Social media and outside funding have given them much more influence today but they’re still kooks.

    Like

  2. Interesting. I’ve been wondering about the hysteria over data centers, especially the idea that somehow any water they used for cooling would somehow magically be destroyed. Which seems to be a common thought among those pushing to stop them. Didn’t seem to gel with what I know about how cooling systems usually work.

    Liked by 1 person

    1. I’ve been confused about that too. I’m like “the only waste is heat”. I’ll admit I hadn’t considered things like piping types.

      Like

  3. There are not sufficient payoffs to NGOs to make it solved problem. Only when the NGO grift from it being solved exceeds the grift from protesting about a problem is it solved.

    Once you realize that the reason 50 years of post Apollo tech and the computer revolution gave us Facebook et al (which make NGO protests louder thus increasing the grift) instead of Moon cities (which provide no obvious NGO grift) makes complete sense.

    Our economy was “Cold War grift” then “NGO grift” for my entire life. I can’t imagine what it is like to live in an economy centered on businesses whose principal goal is pleasing customers.

    Liked by 2 people

  4. Some municipal water treatment facilities add a mix of chlorine and ammonia to the water, because the combination is less toxic than the higher levels of chlorine alone that would be required to get the same results.

    Like

  5. The more I find out about municipal water systems, the more I rejoice in being on a well. It’s our own personal well, too, not a shared well. (Shared wells usually are doped up with additives like chlorine.) I was around ten or twelve when I found out that most city water comes from reservoirs. SURFACE water. Ew, yuck.

    Liked by 1 person

    1. Hmm, my experiences are a bit different. Oregon has a shared well law that permits up to three parties on a well. (And that makes for some interesting problems.) No special treatment is required. We found it to be a Really Good Idea to drop our ownership in the shared well and to do our own.

      The suburb we lived in when I was a kid used a deep well for the town. Apparently it was from an aquifer somehow connected to Lake Superior (stretching down to/past northern-ish Illinois). Chicago used Lake Michigan water. My grandparents lived there, and their tap water was less than tasty. (I much prefer the lime/iron sulfate mix in our well. Living in a geothermally active location warps the tastebuds.)

      When San Jose switched from chlorine to the clor-amine mix, they warned that fish tanks water needed extra treatment. Never heard of any problems for cats & dogs.

      Like

      1. And yet when cleaning, we are enjoined from mixing chlorine bleach and ammonia because of the dangers of producing same chloramine gas. I suppose, as with many things chemical, the danger is in the dose.

        Liked by 1 person

    2. LOL. True, unless a critter sniffs out the well, tries to get a drink, and falls in. Our well was covered but a half-mile from the house, so we didn’t check it daily. Our first indication of a problem was usually when my father got sick. He tended to have the more sensitive stomach I guess.

      Like

  6. I am in agreement with the author’s discussion and conclusions. I have been in the wastewater business for the last 40 years and have seen the changes for the better. There is no doubt that the AI systems will need more water to operate but the treatment of that water will be no different than what comes from industrial wastewater and it will in all likelihood be easier to treat as it will be less contaminated.

    Liked by 2 people

  7. “The on premise power plants fuse the hydrogens in water into helium, and the oxygens into methamphetamine and fentanyl.” is a legitimate parody, and a test question.

    For any contentious set of claims, you can use around six questions to sort the performers and the bots from people interested in and capable of rational discussion.

    Take four real position statements sources from the wild, add one bullshit statement that serves the anti side, and one bullshit statement that serves the pro side.

    Okay, real people interested in rational discussion may be offended at being asked to address the bullshit statemetns. Then they may self select out of responding. But, real rational people can explain that the bullshit statements are bullshit, and make some sort of other summary of the other four.

    In this case, we definitely are not doing stuff with the nuclear forces in hydrogen atoms, and I am not sure that anyone is showing that anything is being done with the chemical bonds (1) in the water molecules.

    As a general rule of thumb, AOC being in favor is a reason to refuse to consider adopting a policy. But, people at times would get annoyed at people following such a rule.

    “God created man in His image” is the major reason not to be shooting AOC, environmentalists, and some of the people handwringing about environmental impacts of data centers and rocketry. “In the image of God” may be equivalent to this human dignity stuff that I would guess that the Pope wrote about recently.

    I don’t personally know how much compute I need for surveillance. But, machine learning/neural nets are about the most expensive possible ways to do it. In general, in the remote sensing world, we use statistical signal processing, or signal processing algorithms (2). There are a ton of papers on ‘akshully, neural nets do this well’. But, the more conventional surveillance algorithms are domain specific, and there are maybe at least half a dozen domains of surveillance. My feeling is that most dudes form their impression of this stuff from Hollywood, and from what the PRC wants people to think. Not from personal experience in sensing.

    I have just enough relevant experience to guess that some things are hard and “press to doubt”. (Now, many people do know me enough to understand that I also catastrophize and convince myself that easy tasks are impossible. This, and not having a real life, is why I had time to study sensing in the first place. So the true difficulty can be wildly easier or actually more difficult (3) than I estimate.)

    Anyway, I don’t think that anyone in the PRC, even their surveillance bros, actually understands how effective their surveillance regime really is. They are totalitarians, and they are doing a complicated technical task relating to regime security or other interest. They are doing a software task, that can fail in ways that are not physically visible to everyone. I would expect that pencil whipping tests is almost mandatory.

    (1) Okay, I do not distinguish enough between covalent bonds and such things as ionic bonding and van der waals forces. Water is complicated, but.

    (2) I do imagine that these could be cheap enough that legitimate US needs could have been accomplished quietly by a rounding error in compute years ago.

    (3) “Oh, I want to finish this, and restart my career before I n X 10 years old” is darkly hilarious in hindsight. The details say a little more than I am comfortable sharing.

    Liked by 1 person

    1. My feeling is that most dudes form their impression of this stuff from Hollywood, and from what the PRC wants people to think. Not from personal experience in sensing.

      All I have to do is attend our training classes and see what the sales weasels are pushing (successfully, by internal metrics). It really doesn’t matter if it works; it’s being USED for that, and we’ve already waited long enough putting guardrails around how prosecutors can use it against us.

      Liked by 1 person

          1. I thought I might have missed one. I got 26. (A-Z copies? Guess not.)

            Ah, what’s life without WP to add a wee bit of chaos? (“Better” is a valid answer.)

            Liked by 2 people

            1. Ah, but I was the poster. That would also explain why I get two copies of every reply and like: One for me as a subscriber, and one for me as the original poster.

              Liked by 1 person

    1. You used the word “offedded”, which offedse WordPress (Deledda Est!) It doesd’t like “defedse” either. Bary Catelli got eved got bodded for a post that bedtio’d havig to “fedd off” codtedtious idteractiods.

      Whed bedtiodig offedse or defedse, odly the adedoidal cobbedts bay be subbitted udboderated. Dabbed daddy-state!

      Liked by 1 person

      1. LOL! I’ve gathered over time that the particular three-letter combo at issue has been blacklisted on this blog specifically because a certain unusually insane and prolific troll goes by a plethora of names that almost always involve it, and the benefit of blocking 95% of that obsessive gibbon’s poop-flinging outweighs the burden of manually releasing the by-catch.

        Liked by 1 person

      1. My favorite take on that format, quite dated now

        Fat Nikita’s

        Getting stronger

        You can’t afford

        To wait much longer!

        — Sherman Tanks

        Liked by 1 person

        1. From 1980-ish, Hwy 395 North, approaching Reno, NV:

          The man who drives

          Half asleep

          Now lies buried

          Six feet under.

          (Which inspires me to ask a blog fulla writers: is there a term of art for that kind of bait-and-switch non-rhyme? Seems like the kind of thing that should have a name.)

          (Also, O Blog Fulla Writers: Is there a term for those “said book” games, like “‘Shut up,’ he explained” or “‘Hello,’ he lied”?)

          Like

  8. Part of the problem is that half or more of the population can’t understand this article. That definitely includes politicians. They shouldn’t be in the decision loop, but here we are….

    Like

  9. “What about using your bathtub for storing water in advance of a predicted storm or natural disaster?”

    I have something called a Water Bob which you put into your bathtub and fill up with water in advance of the predicted storm or natural disaster. You can get one on Amazon for $35. Haven’t had to use it, yet.

    Liked by 1 person

    1. We’d need that because our tub stopper method doesn’t work. Eventually the tub/showers are getting upgraded/replaced with showers. Then we’d need one because no tub (we are not selling or moving. Kid can figure it out later. We have a small hot tub for soaking). Besides, storing water in your tub isn’t for drinking water, could be, but it is for scooping out for flushing.

      For drinking water the Water Bob makes sense to have one not only in bathtub, but in one’s car trunk or pickup bed (for evacuations). 100 gallons of water is a bit difficult to move. The five gallon totes are hard enough. Which BTW, the five gallon totes were always filled up with … Yep, a garden hose.

      Another source of emergency drinking water is direct out of your hot water tank, provided you have one (given options of tankless hot water). Most people won’t think of that either. That hot water comes out of your faucets.

      Regarding municipal water sources. Willamette Valley has small towns where the joke is “buy your white towels, tan/beige, because that is what they will be after they are washed”. Because the mineral content that comes out of the treated well water is so high. Might be slightly better now since both sources are co-mingling with EWEB, EPUP, or SPUP sources out of the Willamette and McKenzie river systems. Case of “actually use it, or lose it” reserved capacity for the utilities that expired if not used. The small towns needed more water to allow any additional growth.

      Like

      1. Checklists for immediate post-earthquake actions, once sure no structures are going to fall on your head, include shutting off water supply valves into your house so potentially contaminated municipal water does not contaminate what’s already in your house, such as in the tank water heater.

        Like

      2. Water Bob is more for “hurricane warning” or “tornado alert” or “kaiju warning” when you get a heads up in advance so you can react and fill it up. In the zero-warning earthquake or tsunami cases you’d best have long term water storage sussed out in advance, with a planned rotation schedule (write dates on clear packing tape on the outside of the container so you don’t run out of container for your notes).

        Though water does not “go bad” per se, the bleach or other treatment you meter in when filling containers in can in effect “go away” so it’s best to dump it out on the flower beds and refill/re-treat on a regular schedule. Biannually often works fine if you are using dedicated water storage containers which are not transparent, but the recommendation for transparent liter bottles used for water storage is to rotate that water every 6 months.

        Like

  10. To the varied data center use point: Per the link I posted the other day (https://cleanview.co/data-centers/us) the highest power capacity currently operational (not “planned”) data center in the U.S. is the 750 megawatt IREN Childress facility in Childress, Tx, which is used for… bitcoin mining.

    Like

  11. Only comment is about drinking water from sink vs toilet tank vs toilet bowl..

    Toilet tank no problem.

    Though I wouldn’t be happy and under normal circumstances would refuse I would drink out of a cleaned toilet bowl before you could ever convince me to drink out of even a cleaned kitchen sink. It would have to santitized to within an inch of my life before I would put anything that had ever been in it into my mouth.

    Kitchen Sponges: Up to 200,000 times more bacteria than a toilet seat.Kitchen Sink Drains: High levels of E. coli and foodborne bacteria.Cutting Boards: Often carry more fecal-based bacteria than a toilet bowl.Toilet Bowl: Relatively fewer bacteria due to regular use of bleach and disinfectants.

    Liked by 2 people

  12. Water from the tap? Oh noes! I have a 60-gallon tank for drinking water delivered from an actual local mountain source, not the Arrowhead (we still use the name, but it’s tap water that we’ve processed) style stuff. Mostly that’s because tap water in San Diego is at the tail of a thousand mile plus journey from the mountains in Colorado, and is so “hard” that you could pound nails with it. In high school chemistry we actually did the experiment, demonstrating that you couldn’t push enough electricity through distilled water to power a tiny light. We tested it on bottled water, and you could (which is a good thing. You do need some minerals.) We tried water from the drinking fountain, and, I swear, it lit a 60 watt light bulb! Also my wife the chemist assured me that San Diego is one of two locations in the US that the tap water actually leaches lead from lead pipes. (Don’t worry, we’ve replaced them all with plastic piping now.)

    Like

    1. Zero water pitchers come with (or use to) a tool to prove the level of contaminates in water before going through the filter (> 0) and after (0). Also shown in commercials to show Brita filters do not filter out everything. The problem these days is getting the Zero water filters (not inexpensive). Surprisingly, EWEB (our utility) water is, while not zero, very low (which just meant the filters lasted a lot longer).

      Like

      1. We’re using Epic filters, also expensive but seems to help my digestion. Epic’s website has a zip code search which lists the official contaminants in your water.

        All I can really say is, one, the water from the filter tastes good; and two, the water in the top of the filter pitcher is straw colored, and what comes out the bottom is clear.

        Like

  13.   Right now, there is a democrat effort underway to block testing for birth control hormones and abortifacients in municipal water supplies, because they say “it’s a witch hunt to demonize and ban birth control, and couldn’t possibly be at a level that is harmful.” (Weaponizing Water: How the Campaign Against Medication Abortion Co-opts Environmental Policy | Guttmacher Institute)

    Oooh, that is interesting.

    Note that milk is still sold with comments that the cattle didn’t get hormones, and these folks want to block even looking at if drinking water has them?

    Liked by 2 people

    1. Point of historical interest: they started out screaming about the effect on humans but couldn’t get any traction (partially because the studies weren’t there) so they switched over to the health of the cows.

      Like

    2. If the hormone levels “can’t possibly be high enough to be dangerous,” why don’t you just run the test? It will confirm your statement…if you’re telling the truth.

      Like

  14. I hadn’t heard any of this nonsense until a friend started bleating about the contaminated water leaving the data centers “not even water anymore.” Huh?

    On the other hand, municipalities ought to make sure these things don’t negatively impact their citizens. Using millions of gallons per day in a desert is just stupid, and making the citizens foot the bill is negligence at best.

    Like

    1. Yep. Not even water anymore – AI data centers are well known for turning it into wine.

      Chardonnay, to be specific. In boxes.

      Like

Leave a comment