Chia Database upgrade – process and results
One of the biggest upgrades in the recent Chia Blockchain 1.3 beta that was just released is the ability to upgrade the v1 Chia Database to v2. The promise here was pretty extreme with Chia promising “approximately 45%” space reduction. That’s quite the promise, so let’s see if it holds up.
First, don’t do this on your main farming rig. I have no idea what edge cases might cause catastrophic issues. But for those willing to take the risk, the process is pretty simple. Just open up a terminal or command prompt, navigate to your Chia CLI in
\chia-blockchain\app-1.2.12269\resources\app.asar.unpacked\daemon (Windows) and type out chia db upgrade. That will kick it off.
The process is very IO intensive, but not particularly hard on the CPU or memory. It runs while your farming runs and didn’t seem to cause any issues for me. I am having an unrelated issue due to the upgrade involving my plot NFTs not getting picked up properly, but that was not related to the DB upgrade. It didn’t fix it either though.
The process took about 37 minutes total and when it was done it left both DB images, so you will have to manually clean up. It runs over 5 phases, each copying a specific section of the database. And what are the results? See for yourself.
Its about 45%. The process reduced my DB from just under 50GB to just under 27GB. This is fantastic and makes farming to small single board computers like the Raspberry Pi possible again. I haven’t made any performance measurements so I have no idea if the new database format lives up to the modest improvements yet, but we should see soon if my farm is out of sync constantly. But I doubt it. Great work, Chia Network.
Related
13 thoughts on “ Chia Database upgrade – process and results ”
What is the relation of this smaller Chia v2 database and the BlueBox compression project? They both aim at smaller DB files, right?
I got the latest chia version on Ubuntu (1.2.12.dev1+gf2a334efb), but when I try to use command to convert database to new version, I got Error: No such command ‘db’. What can I do?
You need version 1.3 currently in Beta.
What changes need to be made to config.yaml for the wallet after the chia db upgrade? it still points to v1
I’ve changed it to v2 for wallet as well. I’m not sure, if it is necessary though.
My database upgrade took “forever”… Over 24 hours!
I upgrade directly on my farming machine which is a Raspberry Pi 4 with 4GB RAM, an SSD for the Chia database and 4GB swap space on the SSD.
Not sure why it took so much longer than yours but in the end it finished… Farming in parallel went smoothly the whole time (no stale partials), except around 1 hour before the end where my farmer suddenly went offline.
For anyone interessted, here is my upgrade log:
opening file for reading: /mnt/ssd/.chia/mainnet/db/blockchain_v1_mainnet.sqlite
opening file for writing: /mnt/ssd/.chia/mainnet/db/blockchain_v2_mainnet.sqlite
initializing v2 version
initializing v2 block store
peak: ca4351781d8b28e784e4d0dacdd994c505e93602a6d93e3dc6d9dbd7e62ea862 height: 1707861
[1/5] converting full_blocks
5791.56 seconds
[2/5] converting sub_epoch_segments_v3
499.42 seconds
[3/5] converting hint_store
350.51 seconds
[4/5] converting coin_store
77411.60 seconds
[5/5] build indices
block store
coin store
hint store
4854.99 seconds
updating config.yaml
database_path: db/blockchain_v2_CHALLENGE.sqlite
LEAVING PREVIOUS DB FILE UNTOUCHED /mnt/ssd/.chia/mainnet/db/blockchain_v1_mainnet.sqlite
Bless you for posting this in a format that is not a Youtube video. I do not need a bloody video to tell me how to run one command. You have a new fan and follower.
When you say you’ll have to manually clean up, could you elaborate? Is is just deleting the one file (blockchain_v1_mainnet.sqlite)?
You can actually delete two files, one db/blockchain_v1_mainnet.sqlite and second one wallet/db/blockchain_wallet_v1_mainnet_xxxxxx.sqlite.
Could you please advise how to upgrate db on Mac?
How is your new database so small? Mine was about 105 GB probably partially because it was so old it had some garbage in it. So I deleted the database and wallet directory in my installation and then restarted the client, running 1.3.3. It then rebuilt the database in 3 days. Now the database file size is about 65 gb. How is yours so much smaller than mine? Granted this article was written a few weeks ago so obviously yours is larger now but not that big??
I mean your article was written 2 months ago, but still.
because he is using V2 update your database it will be half the size at least
Инструкция по уменьшению размера блокчейна Chia в 2 раза
Запустим клиент и синхронизируем с сетью. Если у вас по какой-то причине, включился Wallet режим кошелька, а не Farming, то нужно переключить ее в Farming.
В правом верхнем углу, шестеренка и переключаем опцию mode в положение Farming.
Обновление базы Chia и уменьшение размера в 2 раза.
Сделаем резервную копию папки .chia, в ней находятся блокчейн Chia и кошельков.
На Linux Ubuntu она находится по пути
После резервной копии нам нужно перейти в среду управления chia из консоли.
Ну и вводим команду
Начинается процесс уменьшения размера базы данных. На core i9 процесс занял не более 30 минут. Процесс сжатия происходил в 5 этапов, в конечном результате мы получили новый файл базы данных с именем blockchain_v2_mainnet.sqlite1. Давайте сравним их по размеру.
blockchain_v1_mainnet.sqlite1 — 41 468 мегабайт
blockchain_v2_mainnet.sqlite1 — 23 585 мегабайт
Процент сжатия 43,12% отличный результат. Учитывая, что большинство форков берет обновления от основного клиента Chia, в скором времени поможет не плохо с экономить место на жестком диске.
Как обновить chia до новой версии
Official installer links can be found on chia.net/download and this wiki page.
To install the chia-blockchain, follow the instructions according to your operating system. After installing, follow the remaining instructions in the Quick Start Guide to run the software. You should read the release notes and the wiki/repository FAQ.
| Jump to: | Windows | MacOS | Ubuntu | CentOS / RHEL / Fedora | WSL2 | Amazon Linux 2 | Other platforms |
|---|
| Install from source | Update from source |
|---|
All keys and plots from version prior to Beta 8 (released July 16, 2020) are deprecated and can be deleted. Plots from both Beta 8 and newer should work on mainnet.
The minimum supported specs are that of the Raspberry Pi 4, 4GB model
- Quad core 1.5Ghz CPU (must be 64 bit)
- 4 GB Ram
- Starting with Chia version 1.3.5, Python 3.10 is supported. The minimum supported version is 3.7
- For Chia versions prior to 1.3.5, only Python 3.7, 3.8, and 3.9 are supported
Drive format support
Chia plot files are at least 108GB in size (for K32). To plot successfully requires drives formatted to support large files; e.g. NTFS, APFS, exFAT, ext4, etc. Do not use drives with FAT formatting (FAT12, FAT16, or FAT32) or else plotting will fail. Future versions of Chia will check for unsupported drives, but for now it’s up to each user to check their drive format.
Sleep kills plots
The Chia plotting process may take many hours to complete. If the computer or hard drives go to sleep during the plotting process, the plotting fails and you will need to start over. Please ensure all sleep, hibernate and power saving modes for your computer and hard drives are disabled before starting the Chia plotting process. In the future, Chia will have a resume plot feature. In the meantime, if you do get a failed plot, delete all *.tmp files before starting a new plot.
Updating from Release Candidate to 1.0:
Keys and configs from RC3 and newer should automatically migrate. For more details, read the FAQ. No testnet/TXCH coins migrate to mainnet. Mainnet coins are forever, however.
Install the Windows installer — Chia Blockchain Windows
As the Chia code signing certificate is new you will likely have to ask to keep the download and when you run the installer, you will have to choose «More Info» and «Run Anyway» to be able to run the installer. There is no need to use the command line. Some Windows anti-virus applications are seeing the download as a false positive. You can see the entire source code and build method here so we think it’s safe for you to ask those tools to ignore it. Running the installer while plotting on a previous version will stop your plotting process, so be careful.
You can now proceed to the Quick Start Guide
macOS Mojave (10.14.x) or newer is required.
Install the macOS .dmg installer — Chia Blockchain MacOS
When the installer first runs it will import or create multiple keys and add them to the macOS keychain. You may be prompted up to 3 times for your password. We suggest choosing «always allow.»
You can now proceed to the Quick Start Guide
Install from source — macOS
To build a development version, make sure brew is available before starting the setup and that python 3.7 or newer is installed.
See install from source for detailed instruction.
You can install Chia Blockchain in different ways, depending on your needs:
Set up Chia’s repositories and install from them, for ease of installation and upgrade.
Download the deb package and install it manually and manage upgrades manually.
Install from source
Install using the repository
Before you install Chia Blockchain for the first time on a new machine, you need to set up the Chia repository. Afterward, you can install and update Chia Blockchain from the repository.
Set up the repository
- Update the apt package index and install packages to allow apt to use a repository over HTTPS:
- Add Chia’s official GPG Key:
- Use the following command to set up the stable repository.
Install Chia Blockchain
- Update the apt package index and install the latest stable version of Chia Blockchain
Install from deb files
We have GUI and CLI Only installers available for Ubuntu 18.04 and newer and Debian Buster and newer. There are also arm64 versions of the debs available for Ubuntu and Debian/Respberry Pi OS 64. When using the GUI installer, binary CLI tools can be found in /usr/lib/chia-blockchain/resources/app.asar.unpacked/daemon/
Install from source — Ubuntu
See install from source for detailed instructions.
You can install Chia Blockchain in different ways, depending on your needs:
- Set up Chia’s repositories and install from them, for easy of installation and upgrade
- Download the rpm package and install it manually and manage upgrades manually
- Install from source
Install using the yum/dnf repository
Before you installer Chia Blockchain for the first time on a new machine, you need to set up the Chia repository. Afterward, you can install and update Chia Blockchain from the repository.
Set up and install using yum
- Install the yum-utils package (which provides the yum-config-manager utility) and set up the stable repository.
- Install Chia Blockchain
Set up and install using dnf
- Install the dnf config-manager util and set up the stable repository.
- Install Chia Blockchain using dnf
Install from rpm files
We have GUI and CLI Only installers available for RHEL/CentOS 7 and newer and Fedora 28 and newer. When using the GUI installer, binary CLI tools can be found in /usr/lib/chia-blockchain/resources/app.asar.unpacked/daemon/
Fedora 31+ requires libxcrypt-compat to be installed on the system.
Install from source — CentOS/Red Hat/Fedora
See install from source for detailed instruction.
Install from binary wheels
You can run chia-blockchain in Ubuntu 20.04 LTS via WSL2 on Windows.
NOTE: WSL2 plotting is currently only slightly faster than plotting on the native windows client. WSL2 requires significant tweaking to set up correctly. If you find that daunting, it’s probably easier to just use the native windows client.
You can not run the GUI as WSL2 doesn’t yet support graphical interfaces from WSL2.
Check if you already have WSL2 or WSL1 installed:
From PowerShell, type:
If you get a listing of help topics for wsl commands, you have WSL1, and need to upgrade. To upgrade, follow the instructions here. If you get a blank result or a listing of installed Linux versions, you have WSL2 and are OK to proceed.
If WSL is not installed:
From an Administrator PowerShell:
You will be prompted to reboot.
Installing a new WSL2 instance:
Install Ubuntu 20.04 LTS from the Microsoft Store and run it and complete its initial install steps. You now have a linux bash shell environment that can run linux native software on Windows.
Install from source — WSL2
See install from source for detailed instruction.
Running a standalone Windows wallet gui is deprecated but may return in later versions. You can run the Windows version and share keys. You can also plot in WSL2 and migrate the plots to a Windows farmed plot directory.
Increasing the WSL Maximum Storage Capacity
WSL2 uses a Virtual Hardware Disk (VHD) to store files, and it automatically resizes as files grow. However, the VHD has an initial maximum size of 256 GB. Therefore, the default WSL2 VHD is probably only capable of plotting k=30 plots. To plot anything larger, you will need to increase the maximum allowable size. Follow the guide here.
Setting a maximum limit to WSL2 memory access
If you try plotting Chia in WSL2 without limiting the memory access, WSL2 will use 100% of your available machine’s memory, and your computer will get bogged down and begin swapping memory to your hard drive. This will severely cripple your plotting speeds. To set the maximum memory that WSL2 is allowed to use, create a configuration file as described in this guide.
WSL VHD Plotting Nuances
Plotting within WSL2 can write to either the native VHD (which is EXT4) or to any other drive, which can be NTFS or any other FS-type. Writing to the native VHD is faster than writing out to another drive.
Plotting uses three commands for directory control:
-t for initial temp directory. Phases 1 and 2 happen here.
-2 for secondary temp directory. Phase 3 (compression) happens here.
-d for final destination. Phase 4 happens here.
Plotting works such that -t and -2 require the exact same amount of storage space. Therefore, if -t and -2 point to the same drive, that drive needs 2x the final file size + 1x the max working file size.
For maximum speed, -t and -2 should be inside the WSL2 filesystem. Something like: -t
/chia_temp . Just beware that the WSL2 VHD will need a much larger maximum capacity.
-d can point to any other drive for the final destination.
Install from source — Amazon Linux 2
See install from source for detailed instruction.
Install from binary package
Other install methods and environments
You need Python 3.7 or newer.
Chia strives to provide binary wheels for modern systems. If your system does not have binary wheels, you may need to install development tools to build some Python extensions from source. If you’re attempting to install from source, setting the environment variable BUILD_VDF_CLIENT to N will skip trying to build Timelord components that aren’t very cross platform, e.g. export BUILD_VDF_CLIENT=N .
Create a virtual environment
Your installation goes inside a virtual environment.
There are lots of ways to create and manage a virtual environment. This is just one.
Wheels can be in source or binary format. Binary wheels are specific to an operating system and python version number. Source wheels require development tools.
Chia hosts some binary wheels that are not available from PyPI. This step is optional, but it may succeed where building from source can take a while or fail in hard-to-debug ways. If wheels are not available for your system, this step will fail. But you can try it anyway.
Before you use chia-blockchain in future, you must «enter» your virtual environment.
Install from source
*This instruction is for MacOS, Ubuntu, CentOS, RedHat, WSL2, Amazon Linux 2 and possibly *NIX like OSes.
Update from source
*This instruction is for MacOS, Ubuntu, CentOS, RedHat, WSL2, Amazon Linux 2 and possibly *NIX like OSes.
To Update/Upgrade from the previous version
Sometimes stray daemons left over from previously running processes will cause strange bugs/errors when upgrading to a new version. Make sure all daemons and chia processes are killed before installing or upgrading.
This is normally done by executing chia stop -d all from the upgrade example above.
But it doesn’t hurt to double check using ps -Af | grep chia to make sure there are no chia processes left running. You may have to manually kill the chia daemon if an install and chia start was performed without first running chia stop -d all
If all else fails, rebooting the machine and restarting the chia daemon/processes usually does the trick.
To join a testnet, follow the instructions on the How to Connect to the Testnet article.
It is recommended that you keep a separate testnet environment by prepending CHIA_ROOT=»
/.chia/testnetx» to all of your cli commands. For example, CHIA_ROOT
/.chia/testnet10″ so that all commands will use testnet10 instead of mainnet. If you’re using a version above 1.2.11 (currently in beta), you can update all config values to the testnet values by running chia configure -t true .
Chia новая версия
Расследование показало интересные факты.
Походу это не только 1.1.6, определенно все началось еще раньше.
Win7x64, нода+харвестр с плотами.
После запуска приложения Chia идет массивное неявное выделение RAM на 90% от доступной(8Gb), выглядит как утечка.
Со временем периодически уменьшает потребление до 50% , позже вновь до 90%, и так тудасюда без видимых причин.
После закрытия приложения Chia остается висеть в памяти start_harvester.
Запуск "chia stop all" не помогает, убить процесс нельзя. Умирает сам спустя 10мин, отдавая назад RAM.
После добавления планки +8Gb — все повторяется, отжор 90% от 16Gb. Это не нормально.
Разумеется при таком раскладе плоттер постоянно вылетает за нехваткой RAM.
Харвестер и фармер также вылетают, приложение Chia вешается на вкладке фармера.
Вероятно это одна из причин аномальных нехваток RAM и вылетов плотера, о которых так часто спрашивают.
Удалил из приложения Chia все диски с плотами, примонтировал диски как папки на C:\, добавил папки назад в приложение Chia.
Помогло! Аномальные выделения 90% RAM прекратились.
Есть подозрение, что майнинг все же потребляет RAM, иначе как еще объяснить внезапные скачки RAM (при неактивном плотере).