Everything was good, great in fact. Everything was working, but my OCD weren’t okay with how a few services were set-up, so I cleaned up my yaml, commented my docker compose and felt cushty… right up until it was time to fix Immich.
I have minor beef with Immich and basically any larger project and the way they go about their Docker Compose. Basically I feel they make the assumption that they’re the only thing running.
^Disclaimer: I fully accept this is all just me being too stupid and not the Immich development team.
So first things first, let’s rename database
to immich-database
, redis
to immich-redis
and most importantly, let’s give it a port that’s not the default postgres port that everyone wants to use. Easy right? Nope.
First Immich said it couldn’t find the database, so I went in and added the IP as the DB_URL
to the .env
. But that didn’t really help, so I went back to the Docker Compose and added the path to the references to the env.
Second issue I stumble upon, is that despite the port being available as DB_PORT
Immich decided it was a suggestion and not an instruction. No worries, I edit the database URL to include the port.
Okay, I’m on the home stretch now right. I mean this was working before I decided to mess around with it in the name of scalability or whatever I thought was genius at the time… except
[Nest] 7 - 04/05/2024, 6:10:23 PM ERROR [ExceptionHandler] no PostgreSQL user name specified in startup packet
What does that even mean? Why won’t you work? So I do a web search and everything is saying that docker probably isn’t reading the username from the env file or the Docker Compose. I try adding single quotes and no joy, double quotes, no joy. I have no idea where I’ve gone wrong. I feel like my beautiful simple Docker Compose now looks like Frankenstein’s Monster. Help 🙏
ENV
# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
# The location where your uploaded files are stored
#UPLOAD_LOCATION=immichlibrary
# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
# Connection secret for postgres. You should change it to a random password
DB_PASSWORD="RANDOMLIES"
DB_URL=http://192.168.0.89:8765
DB_PORT=8765
# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
Docker Compose
version: "3.8"
#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#
name: immich
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
command: [ "start.sh", "immich" ]
volumes:
#- ${UPLOAD_LOCATION}:/usr/src/app/upload
- immichlibrary:/usr/src/app/upload
- /etc/localtime:/etc/localtime:ro
env_file:
- /opt/immich/.env
environment:
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_DB: ${DB_DATABASE_NAME}
ports:
- 2283:3001
depends_on:
- immich-redis
- immich-database
restart: always
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
# extends:
# file: hwaccel.yml
# service: hwaccel
command: [ "start.sh", "microservices" ]
volumes:
#- ${UPLOAD_LOCATION}:/usr/src/app/upload
- immichlibrary:/usr/src/app/upload
- /etc/localtime:/etc/localtime:ro
env_file:
- /opt/immich/.env
environment:
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_DB: ${DB_DATABASE_NAME}
depends_on:
- immich-redis
- immich-database
restart: always
immich-machine-learning:
container_name: immich_machine_learning
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
volumes:
- model-cache:/cache
env_file:
- /opt/immich/.env
restart: always
immich-redis:
container_name: immich_redis
image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc
restart: always
immich-database:
container_name: immich_postgres
image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
env_file:
- /opt/immich/.env
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
ports:
- 8765:5432
volumes:
- /opt/immich/postgres:/var/lib/postgresql/data
restart: always
volumes:
model-cache:
driver_opts:
type: "nfs"
o: "addr=192.168.0.245,nolock,soft,rw"
device: ":/mnt/Shared Pictures/.Immich/cache"
immichlibrary:
driver_opts:
type: "nfs"
o: "addr=192.168.0.245,nolock,soft,rw"
device: ":/mnt/Shared Pictures/.Immich"
Even if you read this and don’t feel you can help or have nothing to add, thanks for sharing your time with me 🥹
I have minor beef with Immich and basically any larger project and the way they go about their Docker Compose. Basically I feel they make the assumption that they’re the only thing running.
^Disclaimer: I fully accept this is all just me being too stupid and not the Immich development team.
This might be my turn to be too stupid but isn’t the point of docker that they all run in containers so it doesn’t matter? They can all use the same database port, because the database is in a container and so doesn’t prevent another database container using the same port. The port doesn’t need to be exposed to the host.
The only issue that comes up when running lots of services is accessing them all over http, and that’s what a reverse proxy is for. I run a dozen services on the same machine, mostly using the default docker compose files, and never have to mess with things like you have here.
Yes.
Thank you
Correct yes, each compose project is isolated on its own network as well.
Thank you.
Ooh. I was clearly overthinking it. Thank you so much for dispersing my stupidity.
Same, I have multiple services running in some machines and I’ve never had the need to modify the ports inside each docker-compose network. Just the exposed ports are the ones I’ve changed, and just for integrating with external services or the reverse proxy.
Thank you.
So first things first, let’s rename database to immich-database, redis to immich-redis
Docker compose does this for you, so a service named
database
becomesimmich-database
if your compose project is named “immich” by placing it inside a folder by that name. You would have ended up withimmich-immich-database
Same goes for your volumes.
most importantly, let’s give it a port that’s not the default postgres port that everyone wants to use. Easy right? Nope.
Since the postgres container is on that specific compose project network, the port is not shared and won’t interfere with any other compose projects, so there’s no need to change it away from the default. The main thing is each compose project is isolated by itself.
Also get rid of the
ports:
map for the database, it’s already part of the same compose network and does not need a port exposed to the outside.so I went in and added the IP as the DB_URL to the .env
Don’t use IPs to refer to docker containers as they change, instead use the container name so
database
for example. (Just don’t prefix it with the project name, so don’t useimmich-database
for example).Essentially just undo your changes and it should all work as expected, and will be entirely isolated so won’t interfere with any other containers or compose projects you run.
Thanks for the info. In my head, I had one eye on the future, because honestly, when I chuck it in the corner, I know myself, I won’t be arsed enough to jump from directory to directory to run the individual compose files and so I’m kinda aiming to have everything run from a single docker compose. Am I being naive?
Definitely don’t put everything in a single docker-compose file. For one compose files change from the developer to add/adjust things. You want to make updating to the latest one as easy as copying it over the existing one with minor changes if any, ideally none. For another, you shouldn’t rely on running any of it manually. If you want to chuck it in a corner, write some systemd unit files to start and restart the docker compose services for you. Finally, it’s not too difficult to find what compose files you have once you’ve forgotten them. Use
docker compose ls
, plaindocker
and if all else failsfind
,grep
,ps
. Also you can always write what you did and how to find it in a README.md file. Print it and stick it to the box if you have to. The point is to leave breadcrumbs and be able to follow them, not remember shit you can’t possibly remember for a long time.Read more about docker and compose. They’re extremely powerful and some of the most important tools out there. They’re also very easy to use but you can’t just figure them out beyond the absolute basics, you have to RTFM for how to do things. If something looks difficult, read to see what’s the best way to do it. Chances are it isn’t. At some point you’d be wondering how you’ve lived without docker. 😂
I love this community. Thank you so much.
Yeah you definitely want each ‘thing’ in its own compose file. Take a look at Portainer or Dockge to easily manage multiple compose files.
I don’t understand why you even change the names and ports. If you have a seperate
docker-compose.yml
file for Immich, the names won’t clash with other services (except ifcontainer_name
is duplicated, but services like postgres and redis normally get one assigned automatically).The ports are also limited to the container networks, so running several postgres instances still allows all of them to use the default port (except you pass them through from the host, which you normally shouldn’t do in closed networks like Immich’s or you run all services in
network_mode: host
, which is often a bad idea).Opening ports in a postgres instance is not always needed, because you can attach yourself to the container and use the cli interface to do what you need.
Also the port you opened to change the default port is only for external services or clients. Immich-server uses the internal network for connecting to postgres, which still uses the default port. You should just use
immich-database:5432
and not change anything.Thank you so much. This is something I feel like I should’ve known but is totally new to me. I appreciate it.
They are the only thing running in that container/compose environment. That’s one of the main points of containerization! If you need to connect it to something else, like a reverse proxy, have it run in its own compose network and connect the two networks together.
Thank you, I appreciate you.
RIP
I sympathize with the dissatisfaction with container and service naming schemes - it does get annoying when you take an overall look at all the containers and services running and things get muddy. In Immich, I renamed my services and containers for overall clarity. But why mess with ports?
I had a master plan to combine all my compose files into one so I could just do one pull and one up every morning while I drink my tea 🥺
Best practice is to keep each project in its own folders with their respective docker compose file in there.
Later, on your journey you will want more granular control especially with networks and reverse proxy setups.
Oof, that’s gonna be my next project as I spin up my own router 🤤
Something I’ve noticed that is somewhat related but tangential to your problem: The result I’ve always gotten from using compose files is that container names and volume names get assigned names that contain a shared prefix by default. I don’t use docker and instead prefer podman but I would expect both to behave the same on this front. For example, when I have a file at
nextcloud/compose.yml
that looks like this:volumes: nextcloud: db: services: db: image: docker.io/mariadb:10.6 ... app: image: docker.io/nextcloud ...
I end up with volumes named
nextcloud_nextcloud
andnextcloud_db
, with containers namednextcloud_db
andnextcloud_app
, as long as neither of those services overrides this behavior by specifying acontainer_name
. I believe this prefix probably comes from the file-levelname:
if there is one and the parent directory’s name otherwise.The reasons I adjust my own compose files to be different from the image maintainer’s recommendation include: to accommodate the differences between podman and docker, avoiding conflicts between the exported listen ports, any host filesystem paths I want to mount in the container, and my own preferences. The only conflict I’ve had with other containers there is the exported port. zigbee2mqtt, nextcloud, and freshrss all suggest using port 8080 so I had to change at least two of them in order to run all three.
Early on in my Docker journey (I’m aware I sound like a total wanker with that, allow me this once 😂) I had to add a bunch of container names to keep things readable and have kept at it ever since.
Hey so I’m intermediate at best. This to me sounds like permissions/routing problems. No user name specified in startup packet?
This sounds to me like you edited something that starts up postgres such that when it is started up it is not receiving a username at all
Shooting in the dark here, but perhaps take a look through logs for postgres. Check postgres and redis configs in these docker containers, they might be set up expecting something you changed, so they’re not receiving anything at all.
You’re sending an unmodified username to a new host name from what I see, right? Is the other side of this connection aware of these modifications?
Fuck that sounds vague. I might not even be right. I’d be digging through error logs for clues.
Like others have said, though. The whole benefit of dockerizing/containerizing services is so that that Docker container can contain anything that might otherwise conflict with it running in your systems environment. So it’s okay to have a doctor container where you’re using some environment variable like “database”. You don’t need to make it “service_database” in that services container.
Hope I am am providing some value here. It’s possible I not 🤷♂️
You truly are providing value and I appreciate you for it. Thank you.
😁
I also asked a local instance of the Mixtral llm:
Database URL Format: The DB_URL in the .env file is defined as http://192.168.0.89:8765, which is unusual for a PostgreSQL connection. PostgreSQL typically uses a connection string format, not a URL starting with http://. The correct format for a PostgreSQL connection string usually looks like: postgresql://username:password@hostname:port/database. Changing the DB_URL to match this format might resolve the issue of the application not recognizing the username.
Environment Variable Consistency: Ensure that all references to the database, Redis, and other services within both the .env file and the Docker Compose file are consistent with the new names and ports. This includes checking that the DB_HOSTNAME and REDIS_HOSTNAME in the .env file match the container_name of the services in the Docker Compose file.
PostgreSQL Connection String in Docker Compose: The environment variables for the Immich services in the Docker Compose file are focused on POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB. If Immich constructs its own connection string internally using these components, ensure they’re correctly configured. However, it might be worth directly specifying a full connection string (if supported by Immich) that includes the username, password, host, and database in a single variable, to reduce the chance of misconfiguration.
Network Configuration: Verify that Docker’s network configuration hasn’t been altered in a way that would prevent containers from communicating properly. Docker Compose services communicate over a default network, but if you’ve made network modifications, ensure that these don’t inadvertently block connections between services.
Thank you so much!
Did ya get it? :)
When you rename a service it helps to figure out why you’re doing it, because there’s different ways in which they relate to each other. There’s usually no point in renaming the services because they’re only used inside the compose for dependencies. You can rename the containers names but you have to be careful because if you don’t specify hostnames explicitly it uses the contain name and the services won’t find each other anymore.
I believe Immich ignores those env vars that specify the db and redis hostnames so don’t mess with that.
Thank you very much. I had one eye on combining all my compose files in the future to make things more remotely manageable and that’s probably been my folly here.
It’s good to want to have that but dumping everything in one compose file is not the way. Read through the compose docs, there are many things you can use to achieve modularity. Start with fragments then continue with the next sections (extensions, interpolation, merge, include and profiles).
If you want to do things with networking you can, there’s a ton of things you can do to make containers connect in all kinds of networks, you can control IPs, you can control MACs, you can make containers show up as regular machines on your LAN and take their IP from DHCP and have names in DNS etc.
The only reason to put things in the same compose file is as a convenience when you have a set of containers that are strongly related and make up a single “app” between them, but some applications go overboard with complexity. Immich is merely mildly annoying, have a look at mailcow for comparison, now that’s bad.
I’m not going to lie to you, clicking that second link was overwhelming, but that first link… that made me grin like a child that just got a bag of sweets. Thank you!
It’s worth reading through all the docker docs sometime, it can do a lot of cool stuff. I’ll leave these here:
The mailcow compose is actually worth coming back to when you’ve had some more experience. There are a ton of interesting tricks in there like overriding the DNS server, defining fallback values for env vars, sharing/unsharing volumes between containers with the
:z
and:Z
flags, [ab]using thecommand
to perform initializations or to wait for another container etc.
Can’t help but I totally agree about projects that assume they are the only thing running. That is the reason I don’t run nextcloud.
What? I’ve never had the feeling that nextcloud assumes that. Are you using a special all-in-one docker image? Because I am using the regular one and pair it with db, redis etc. containers and am absolutely happy with it.
Possibly that is the issue. I try every once in a while to see.
The AIO sucks. To be honest, NextCloud kinda sucks anyway. I’ve had to wrestle with it way more than I’d like.
NextCloud Pi was dreamy. But that got discontinued and they recommended the AIO, which is overkill. I ended up installing normal and even then, you click to not install the recommended and it still installs a bunch where you have to disable things.
On top of that, I feel the installer should create the database too.
The port for your postgres container is still the same for other containers, what you did was just map 5432 to 8765 for your host.
You don’t need to change the port or the host the immich services try to access within the network docker compose creates. You still have
container_name: immich_postgres
so you didn’t change anything for the other containers.What you did was change how to write the command to up or down the container. From
docker compose up database
todocker compose up immich-database
(which normally you won’t use since you want to up and down everything at once).
If you dodocker ps
you’ll still see the name of the container isimmich_postgres
Thank you for the info
Man, I feel you. I hate Mattermost for its utter inability to run on anything else than port 8065 specifically.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters DHCP Dynamic Host Configuration Protocol, automates assignment of IPs when connecting to a network DNS Domain Name Service/System IP Internet Protocol
3 acronyms in this thread; the most compressed thread commented on today has 10 acronyms.
[Thread #659 for this sub, first seen 6th Apr 2024, 12:15] [FAQ] [Full list] [Contact] [Source code]
deleted by creator
I’m listening