Yes familiar with the hardware side of things from pure embedded processors which run no OS at all, up to embedded Linux architectures where pseudo real-time data gathering is performed.
I don't know what to tell you as far as advice, but also know volumes to tell.
@ON-Soapbox
What you're really talking about is embedded programming.
There is a forum on LQ covering that. Not much activity, but here it is
Linux - Embedded & Single-board computer. Further, there are a few groups on LinkedIn covering embedded, not just for Linux.
My degree is Electrical Engineering. I have always worked on embedded devices which are more lower level software, or firmware, that is very closely coupled to the hardware. We work in a world where we measure signals timing and levels to validate that things are happening or that things are correctly hooked up. For instance an interface may need to be configured properly, it may need to be electrically tied high or low to option something to operate a certain way or allow the interface to operate as intended. The pin or interface may be an input or an output with respect to the CPU or some other chip. For things like I2C or SPI interfaces, we pay attention to the signal levels, the timing (clock) as well as the, signal sequence requirements for that interface. From there you next need to worry about the specification for the particular device you're talking too, which means reading the chip spec. For things like serial communications, we worry over how to derive the baud rate from the clock rate and divide down registers, as well as the voltage level for the serial connection. Same for things like a USB interface, electrical, signals, and then the actual driver to talk to it. Next we may worry about interrupts for TX or RX and the associated buffers we need to establish to clock in or out data. For things like A2D conversion there are issues like controlling the cycle time of signals so that we give the A2D enough time to acquire a measurement, and we may have to worry about the precision of the bits we have and whether or not the input signal needs to be scaled somehow to allow for maximum range of measurements within the bit precision we have available; or we may need more bits to represent a valid measurement. Other things might be controlling direct memory access (DMA) controllers, and just generally designing an architecture to be able to fulfill the design needs but be capable of operating. To couple both of those ideas, we had to control the acquire time for a bank of A2D where they dumped their measurements into a DMA. This involved ensuring that we were not accessing the memory and that electrically all was set for the A2C to perform it's writes. And then service the interrupt that indicated that acquisition was complete. Chips have register configuration settings to control all this, but one needs to read up on their spec and understand how it all works. And there's also the situations where the documentation is slightly incorrect and you have to find out via the web, your own experimentation, or customer support forums. There never seems to be enough resources, due to feature creep, you start with a fully capable processor choice, and then the customer adds tons of added wants and features and suddenly you are out of real-time; meaning you cannot fulfill all these tasks in sufficient time and you do not have enough processing bandwidth.
So ... for experimentation. I say, buy a Raspberry Pi and read up on many of the experimental projects where people do stuff like wire up a real-time clock over the I2C and get that working. Then add stuff like a temperature measurement device or a programmable resistor. Stuff which is analog as well as digital. Add things that you talk to via GPIO, like a keypad or some switches. Be aware that as you get more sophisticated, that you'd be advised to have an Oscilloscope and a multimeter, as well as some savvy with soldering electronic components, or at least breadboarding capabilities to get more deeply involved.
And then pick how far you wish to go. Obviously one can go very far, but it grows increasingly complex, and obviously the normal persons run out of initiative and personal horsepower, as you're advertising in your initial thoughts. Obviously working in that field, being required to produce results, and having a team of peers around to toss ideas with helps immensely, as well as the infrastructure where the company owns many scopes, analyzers, and has the capabilities to create electronic boards and such, or just buy development kids in support of our efforts. That can be done on the cheap, just you'll come close to maxing out at some point either with ideas, initiative, dollars, or some combination thereof. And there are people who have unlimited initiative, ideas, etc and they find ways to do their projects. The rare few continue just playing. Mainly the best eventually end up working in a situation which profits them personally, either via salary or if they develop a product which they sell.
Note that Arduino is a processor, and Raspberry Pi is a System on Module which uses an ARM processor.
Another very important point that I've learned over the years: Sorry CS people!
Those with Computer Science degrees know a TON about algorithms, modeling stack size, Turing machines, etc. They do not have training, in general, to inform them about things like Microprocessor design, or the design of an Arithmetic Logic Unit, and therefore they don't understand how instructions are CREATED in silicone. As a result, they are hobbled somewhat when it comes to hardware understanding. We once discussed a finite state machine design and decided to swap it out from one formulation to another, the EE's got it, the CS's did not. But ... then again we were talking about a set of "terms" Moore versus Mealy. These days one can just google that and get it. And further, it wasn't as if the CS people could not understand it, they could easily. But it highlighted to me the difference in training. And when other subjects came up like the electrical signal levels needed for interfacing to a chip, the CS people became very much hands off. Some liked it and went in that direction. So you have to have an inclination to do that stuff and not let minor setbacks halt you, but instead be tenacious and debug something so you understand it. If that's not you, then likely you're not inclined to be strongly involved with embedded programming, but maybe the periphery where most of the connections are standardized and can easily be set up with minimal debug is OK for you.
@OFF-Soapbox