diff options
| -rw-r--r-- | .gitignore | 3 | ||||
| -rw-r--r-- | CONTRIBUTING.md | 223 | ||||
| -rw-r--r-- | LICENSE.md | 29 | ||||
| -rw-r--r-- | LICENSES/CC-BY-SA-4.0 | 428 | ||||
| -rw-r--r-- | LICENSES/MIT | 21 | ||||
| -rw-r--r-- | PROGRESS.md | 542 | ||||
| -rw-r--r-- | README.md | 38 | ||||
| -rw-r--r-- | build.sh | 215 | ||||
| -rwxr-xr-x | build/mo | 2037 | ||||
| -rw-r--r-- | build/templates/home.html | 12 | ||||
| -rw-r--r-- | build/templates/page.html | 12 | ||||
| -rw-r--r-- | build/templates/partials/.gitkeep | 0 | ||||
| -rw-r--r-- | build/templates/partials/footer.html | 16 | ||||
| -rw-r--r-- | build/templates/partials/head.html | 11 | ||||
| -rw-r--r-- | build/templates/partials/header.html | 6 | ||||
| -rw-r--r-- | linux-command-line/shell-basics/index.md | 327 |
16 files changed, 3920 insertions, 0 deletions
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..c1c3231 --- /dev/null +++ b/.gitignore @@ -0,0 +1,3 @@ +.worknotes/ +output/ + diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..7173170 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,223 @@ +--- +title: contributing guide +--- + +# contributing to ubergeek + +ubergeek is a community study guide. Contributions are welcome — fixes, +improvements, new exercises, better explanations, and sample projects. + +The primary repository lives on **sourcehut** and is mirrored to GitHub and +GitLab. You can contribute from any platform. + +--- + +## Ways to Contribute + +- **Fix errors** — typos, incorrect information, broken examples +- **Improve explanations** — clearer wording, better analogies, added context +- **Add exercises and projects** — practice problems, mini-projects, challenge sets +- **Add sample code** — working examples in any of the covered languages +- **Add diagrams and images** — ASCII art, schematics, architecture diagrams +- **Review existing content** — flag sections that are confusing or incomplete +- **Suggest new topics or modules** — open a discussion first + +--- + +## Repository Structure + +``` +topic-name/ +├── index.md # Topic overview +├── images/ # Diagrams, schematics, screenshots (.png, .svg, .jpg) +├── samples/ # Runnable code and project files +│ └── project-name/ +│ ├── src/ +│ ├── Makefile +│ └── README.md +└── module-name/ + └── index.md # Module content +``` + +- Content goes in `index.md` files using CommonMark markdown +- Images go in the topic-level `images/` directory +- Sample code and project files go in the topic-level `samples/` directory +- Every sample project should include a README explaining how to build and run it +- Reference images using relative paths: `` + +--- + +## Content Style Guide + +### Tone +- Direct and practical — teach by showing, not lecturing +- Assume intelligence, not prior knowledge +- Use "you" not "the student" or "one" + +### Structure +- Every page starts with frontmatter (`title` required) +- Open with a short paragraph explaining what and why +- Use code blocks liberally — real, runnable examples +- End modules with exercises (drill, challenges, mini-project) +- End with key takeaways and a "Next" link + +### Code +- All code must be correct and runnable +- Include comments only where the logic is not obvious +- Show output where it helps understanding +- For multi-language topics, show C, Python, Rust, C++, and Bash where applicable + +### Software Installation +- When a module requires extra software, link to the official site and include + installation steps for Arch Linux (primary), Debian/Ubuntu, and Fedora +- Use system package managers where possible +- Note any version requirements + +### Images +- Prefer ASCII diagrams for simple cases +- Store image files in `topic/images/` with descriptive names +- Reference using relative paths: `` +- Always include `alt` text + +### Naming +- Directories: `kebab-case` (e.g., `text-processing`, `pointers-and-memory`) +- Files: `kebab-case.md` for content, descriptive names for images and samples +- No spaces, no uppercase in file/directory names + +--- + +## Contributing via Sourcehut (primary) + +The canonical repository is on +[sr.ht](https://git.sr.ht/~gumxcc/ubergeek). Contributions are accepted via +email patches. + +### Setup + +```bash +# Clone the repo +git clone https://git.sr.ht/~gumxcc/ubergeek +cd ubergeek + +# Configure git send-email (one-time setup) +git config sendemail.to ~gumxcc/ubergeek@lists.sr.ht +git config format.subjectPrefix "PATCH ubergeek" +``` + +### Submitting a Patch + +```bash +# Create a branch and make your changes +git checkout -b fix-typo-shell-basics +# ... edit files ... +git add -A +git commit -m "fix: correct permissions example in shell basics" + +# Send patch via email +git send-email --to=~gumxcc/ubergeek@lists.sr.ht HEAD~1 +``` + +For multi-commit changes: + +```bash +git send-email --to=~gumxcc/ubergeek@lists.sr.ht origin/main..HEAD +``` + +### Discussion + +Use the mailing list for questions, suggestions, and reviews: +`~gumxcc/ubergeek@lists.sr.ht` + +See [git-send-email.io](https://git-send-email.io) for a tutorial on setting +up `git send-email`. + +--- + +## Contributing via GitHub / GitLab (mirrors) + +Pull requests (GitHub) and merge requests (GitLab) are both accepted. + +```bash +# Fork the repo on GitHub or GitLab, then clone your fork +git clone https://github.com/gumxcc/ubergeek.git # or gitlab.com +cd ubergeek + +# Create a branch +git checkout -b fix-typo-shell-basics + +# Make changes, commit, push +git add -A +git commit -m "fix: correct permissions example in shell basics" +git push origin fix-typo-shell-basics + +# Open a pull/merge request from the web UI +``` + +### PR/MR Guidelines +- Keep PRs focused — one fix or one module per PR +- Write a clear description of what you changed and why +- If adding a new module, open an issue first to discuss scope + +--- + +## Sample Code Guidelines + +When adding to `samples/`: + +- **Must compile/run out of the box** — include a Makefile, requirements.txt, + Cargo.toml, or whatever the project needs +- **Include a README.md** — what it does, how to build, how to run, expected output +- **Keep it minimal** — demonstrate the concept, nothing more +- **Pin dependencies** where practical — don't rely on "latest" +- **Test on Linux** — that's the primary platform for this guide + +### KiCad / Hardware Projects + +- Include the full KiCad project directory (`.kicad_pro`, `.kicad_sch`, `.kicad_pcb`) +- Add a README.md with a description and BOM +- Export a schematic PDF and place it in `images/` for people without KiCad +- Use KiCad's built-in libraries where possible + +--- + +## Commit Messages + +``` +<type>: <short description> + +<optional body explaining why> +``` + +Types: +- `content` — new or updated study material +- `fix` — corrections to existing content +- `exercises` — new or updated exercises +- `samples` — new or updated sample code/projects +- `images` — new or updated diagrams/images +- `meta` — PROGRESS.md, CONTRIBUTING.md, index.md, study-plan.md +- `infra` — CI, build scripts, tooling + +Examples: +``` +content: add shell basics module for linux command line +fix: correct voltage divider formula in electronics +exercises: add pipeline challenges to text processing +samples: add bare-metal blinky project for STM32F4 +``` + +--- + +## Review Process + +1. All contributions are reviewed before merging +2. Content accuracy is the top priority +3. Style and formatting can be fixed post-merge — don't let it block good content +4. Disagreements are resolved by discussion, not authority + +--- + +## License + +By contributing, you agree that your contributions will be licensed under +CC BY-SA 4.0, the same license as the rest of the project. Code samples in +`samples/` directories are additionally available under the MIT license. diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 0000000..c78c9c0 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,29 @@ +# License + +This project uses a dual-license model. + +## Content — CC BY-SA 4.0 + +All written content, documentation, exercises, images, and diagrams are +licensed under the +[Creative Commons Attribution-ShareAlike 4.0 International](LICENSES/CC-BY-SA-4.0) +license. + +You are free to share and adapt the material for any purpose, including +commercially, as long as you give credit and distribute derivative works under +the same license. + +## Code Samples — MIT + +All code in `samples/` directories throughout the repository is licensed under +the [MIT License](LICENSES/MIT). + +You are free to use, modify, and distribute the code with no copyleft +restrictions. + +## Which license applies? + +| Path | License | +|------|---------| +| `*/samples/**` | MIT | +| Everything else (`.md` files, images, diagrams) | CC BY-SA 4.0 | diff --git a/LICENSES/CC-BY-SA-4.0 b/LICENSES/CC-BY-SA-4.0 new file mode 100644 index 0000000..2d58298 --- /dev/null +++ b/LICENSES/CC-BY-SA-4.0 @@ -0,0 +1,428 @@ +Attribution-ShareAlike 4.0 International + +======================================================================= + +Creative Commons Corporation ("Creative Commons") is not a law firm and +does not provide legal services or legal advice. Distribution of +Creative Commons public licenses does not create a lawyer-client or +other relationship. Creative Commons makes its licenses and related +information available on an "as-is" basis. Creative Commons gives no +warranties regarding its licenses, any material licensed under their +terms and conditions, or any related information. Creative Commons +disclaims all liability for damages resulting from their use to the +fullest extent possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and +conditions that creators and other rights holders may use to share +original works of authorship and other material subject to copyright +and certain other rights specified in the public license below. The +following considerations are for informational purposes only, are not +exhaustive, and do not form part of our licenses. + + Considerations for licensors: Our public licenses are + intended for use by those authorized to give the public + permission to use material in ways otherwise restricted by + copyright and certain other rights. Our licenses are + irrevocable. Licensors should read and understand the terms + and conditions of the license they choose before applying it. + Licensors should also secure all rights necessary before + applying our licenses so that the public can reuse the + material as expected. Licensors should clearly mark any + material not subject to the license. This includes other CC- + licensed material, or material used under an exception or + limitation to copyright. More considerations for licensors: + wiki.creativecommons.org/Considerations_for_licensors + + Considerations for the public: By using one of our public + licenses, a licensor grants the public permission to use the + licensed material under specified terms and conditions. If + the licensor's permission is not necessary for any reason--for + example, because of any applicable exception or limitation to + copyright--then that use is not regulated by the license. Our + licenses grant only permissions under copyright and certain + other rights that a licensor has authority to grant. Use of + the licensed material may still be restricted for other + reasons, including because others have copyright or other + rights in the material. A licensor may make special requests, + such as asking that all changes be marked or described. + Although not required by our licenses, you are encouraged to + respect those requests where reasonable. More considerations + for the public: + wiki.creativecommons.org/Considerations_for_licensees + +======================================================================= + +Creative Commons Attribution-ShareAlike 4.0 International Public +License + +By exercising the Licensed Rights (defined below), You accept and agree +to be bound by the terms and conditions of this Creative Commons +Attribution-ShareAlike 4.0 International Public License ("Public +License"). To the extent this Public License may be interpreted as a +contract, You are granted the Licensed Rights in consideration of Your +acceptance of these terms and conditions, and the Licensor grants You +such rights in consideration of benefits the Licensor receives from +making the Licensed Material available under these terms and +conditions. + + +Section 1 -- Definitions. + + a. Adapted Material means material subject to Copyright and Similar + Rights that is derived from or based upon the Licensed Material + and in which the Licensed Material is translated, altered, + arranged, transformed, or otherwise modified in a manner requiring + permission under the Copyright and Similar Rights held by the + Licensor. For purposes of this Public License, where the Licensed + Material is a musical work, performance, or sound recording, + Adapted Material is always produced where the Licensed Material is + synched in timed relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright + and Similar Rights in Your contributions to Adapted Material in + accordance with the terms and conditions of this Public License. + + c. BY-SA Compatible License means a license listed at + creativecommons.org/compatiblelicenses, approved by Creative + Commons as essentially the equivalent of this Public License. + + d. Copyright and Similar Rights means copyright and/or similar rights + closely related to copyright including, without limitation, + performance, broadcast, sound recording, and Sui Generis Database + Rights, without regard to how the rights are labeled or + categorized. For purposes of this Public License, the rights + specified in Section 2(b)(1)-(2) are not Copyright and Similar + Rights. + + e. Effective Technological Measures means those measures that, in the + absence of proper authority, may not be circumvented under laws + fulfilling obligations under Article 11 of the WIPO Copyright + Treaty adopted on December 20, 1996, and/or similar international + agreements. + + f. Exceptions and Limitations means fair use, fair dealing, and/or + any other exception or limitation to Copyright and Similar Rights + that applies to Your use of the Licensed Material. + + g. License Elements means the license attributes listed in the name + of a Creative Commons Public License. The License Elements of this + Public License are Attribution and ShareAlike. + + h. Licensed Material means the artistic or literary work, database, + or other material to which the Licensor applied this Public + License. + + i. Licensed Rights means the rights granted to You subject to the + terms and conditions of this Public License, which are limited to + all Copyright and Similar Rights that apply to Your use of the + Licensed Material and that the Licensor has authority to license. + + j. Licensor means the individual(s) or entity(ies) granting rights + under this Public License. + + k. Share means to provide material to the public by any means or + process that requires permission under the Licensed Rights, such + as reproduction, public display, public performance, distribution, + dissemination, communication, or importation, and to make material + available to the public including in ways that members of the + public may access the material from a place and at a time + individually chosen by them. + + l. Sui Generis Database Rights means rights other than copyright + resulting from Directive 96/9/EC of the European Parliament and of + the Council of 11 March 1996 on the legal protection of databases, + as amended and/or succeeded, as well as other essentially + equivalent rights anywhere in the world. + + m. You means the individual or entity exercising the Licensed Rights + under this Public License. Your has a corresponding meaning. + + +Section 2 -- Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, + the Licensor hereby grants You a worldwide, royalty-free, + non-sublicensable, non-exclusive, irrevocable license to + exercise the Licensed Rights in the Licensed Material to: + + a. reproduce and Share the Licensed Material, in whole or + in part; and + + b. produce, reproduce, and Share Adapted Material. + + 2. Exceptions and Limitations. For the avoidance of doubt, where + Exceptions and Limitations apply to Your use, this Public + License does not apply, and You do not need to comply with + its terms and conditions. + + 3. Term. The term of this Public License is specified in Section + 6(a). + + 4. Media and formats; technical modifications allowed. The + Licensor authorizes You to exercise the Licensed Rights in + all media and formats whether now known or hereafter created, + and to make technical modifications necessary to do so. The + Licensor waives and/or agrees not to assert any right or + authority to forbid You from making technical modifications + necessary to exercise the Licensed Rights, including + technical modifications necessary to circumvent Effective + Technological Measures. For purposes of this Public License, + simply making modifications authorized by this Section 2(a) + (4) never produces Adapted Material. + + 5. Downstream recipients. + + a. Offer from the Licensor -- Licensed Material. Every + recipient of the Licensed Material automatically + receives an offer from the Licensor to exercise the + Licensed Rights under the terms and conditions of this + Public License. + + b. Additional offer from the Licensor -- Adapted Material. + Every recipient of Adapted Material from You + automatically receives an offer from the Licensor to + exercise the Licensed Rights in the Adapted Material + under the conditions of the Adapter's License You apply. + + c. No downstream restrictions. You may not offer or impose + any additional or different terms or conditions on, or + apply any Effective Technological Measures to, the + Licensed Material if doing so restricts exercise of the + Licensed Rights by any recipient of the Licensed + Material. + + 6. No endorsement. Nothing in this Public License constitutes or + may be construed as permission to assert or imply that You + are, or that Your use of the Licensed Material is, connected + with, or sponsored, endorsed, or granted official status by, + the Licensor or others designated to receive attribution as + provided in Section 3(a)(1)(A)(i). + + b. Other rights. + + 1. Moral rights, such as the right of integrity, are not + licensed under this Public License, nor are publicity, + privacy, and/or other similar personality rights; however, to + the extent possible, the Licensor waives and/or agrees not to + assert any such rights held by the Licensor to the limited + extent necessary to allow You to exercise the Licensed + Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this + Public License. + + 3. To the extent possible, the Licensor waives any right to + collect royalties from You for the exercise of the Licensed + Rights, whether directly or through a collecting society + under any voluntary or waivable statutory or compulsory + licensing scheme. In all other cases the Licensor expressly + reserves any right to collect such royalties. + + +Section 3 -- License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the +following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified + form), You must: + + a. retain the following if it is supplied by the Licensor + with the Licensed Material: + + i. identification of the creator(s) of the Licensed + Material and any others designated to receive + attribution, in any reasonable manner requested by + the Licensor (including by pseudonym if + designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of + warranties; + + v. a URI or hyperlink to the Licensed Material to the + extent reasonably practicable; + + b. indicate if You modified the Licensed Material and + retain an indication of any previous modifications; and + + c. indicate the Licensed Material is licensed under this + Public License, and include the text of, or the URI or + hyperlink to, this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any + reasonable manner based on the medium, means, and context in + which You Share the Licensed Material. For example, it may be + reasonable to satisfy the conditions by providing a URI or + hyperlink to a resource that includes the required + information. + + 3. If requested by the Licensor, You must remove any of the + information required by Section 3(a)(1)(A) to the extent + reasonably practicable. + + b. ShareAlike. + + In addition to the conditions in Section 3(a), if You Share + Adapted Material You produce, the following conditions also apply. + + 1. The Adapter's License You apply must be a Creative Commons + license with the same License Elements, this version or + later, or a BY-SA Compatible License. + + 2. You must include the text of, or the URI or hyperlink to, the + Adapter's License You apply. You may satisfy this condition + in any reasonable manner based on the medium, means, and + context in which You Share Adapted Material. + + 3. You may not offer or impose any additional or different terms + or conditions on, or apply any Effective Technological + Measures to, Adapted Material that restrict exercise of the + rights granted under the Adapter's License You apply. + + +Section 4 -- Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that +apply to Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right + to extract, reuse, reproduce, and Share all or a substantial + portion of the contents of the database; + + b. if You include all or a substantial portion of the database + contents in a database in which You have Sui Generis Database + Rights, then the database in which You have Sui Generis Database + Rights (but not its individual contents) is Adapted Material, + including for purposes of Section 3(b); and + + c. You must comply with the conditions in Section 3(a) if You Share + all or a substantial portion of the contents of the database. + +For the avoidance of doubt, this Section 4 supplements and does not +replace Your obligations under this Public License where the Licensed +Rights include other Copyright and Similar Rights. + + +Section 5 -- Disclaimer of Warranties and Limitation of Liability. + + a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE + EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS + AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF + ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, + IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, + WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR + PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, + ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT + KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT + ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. + + b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE + TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, + NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, + INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, + COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR + USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN + ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR + DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR + IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. + + c. The disclaimer of warranties and limitation of liability provided + above shall be interpreted in a manner that, to the extent + possible, most closely approximates an absolute disclaimer and + waiver of all liability. + + +Section 6 -- Term and Termination. + + a. This Public License applies for the term of the Copyright and + Similar Rights licensed here. However, if You fail to comply with + this Public License, then Your rights under this Public License + terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under + Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided + it is cured within 30 days of Your discovery of the + violation; or + + 2. upon express reinstatement by the Licensor. + + For the avoidance of doubt, this Section 6(b) does not affect any + right the Licensor may have to seek remedies for Your violations + of this Public License. + + c. For the avoidance of doubt, the Licensor may also offer the + Licensed Material under separate terms or conditions or stop + distributing the Licensed Material at any time; however, doing so + will not terminate this Public License. + + d. Sections 1, 5, 6, 7, and 8 survive termination of this Public + License. + + +Section 7 -- Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different + terms or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the + Licensed Material not stated herein are separate from and + independent of the terms and conditions of this Public License. + + +Section 8 -- Interpretation. + + a. For the avoidance of doubt, this Public License does not, and + shall not be interpreted to, reduce, limit, restrict, or impose + conditions on any use of the Licensed Material that could lawfully + be made without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is + deemed unenforceable, it shall be automatically reformed to the + minimum extent necessary to make it enforceable. If the provision + cannot be reformed, it shall be severed from this Public License + without affecting the enforceability of the remaining terms and + conditions. + + c. No term or condition of this Public License will be waived and no + failure to comply consented to unless expressly agreed to by the + Licensor. + + d. Nothing in this Public License constitutes or may be interpreted + as a limitation upon, or waiver of, any privileges and immunities + that apply to the Licensor or You, including from the legal + processes of any jurisdiction or authority. + + +======================================================================= + +Creative Commons is not a party to its public +licenses. Notwithstanding, Creative Commons may elect to apply one of +its public licenses to material it publishes and in those instances +will be considered the “Licensor.” The text of the Creative Commons +public licenses is dedicated to the public domain under the CC0 Public +Domain Dedication. Except for the limited purpose of indicating that +material is shared under a Creative Commons public license or as +otherwise permitted by the Creative Commons policies published at +creativecommons.org/policies, Creative Commons does not authorize the +use of the trademark "Creative Commons" or any other trademark or logo +of Creative Commons without its prior written consent including, +without limitation, in connection with any unauthorized modifications +to any of its public licenses or any other arrangements, +understandings, or agreements concerning use of licensed material. For +the avoidance of doubt, this paragraph does not form part of the +public licenses. + +Creative Commons may be contacted at creativecommons.org. + diff --git a/LICENSES/MIT b/LICENSES/MIT new file mode 100644 index 0000000..dd99b59 --- /dev/null +++ b/LICENSES/MIT @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 gumxcc and contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/PROGRESS.md b/PROGRESS.md new file mode 100644 index 0000000..abb0cfb --- /dev/null +++ b/PROGRESS.md @@ -0,0 +1,542 @@ +--- +title: ubergeek study guide — progress tracker +--- + +# ubergeek study guide + +Master progress tracker and study plan. Single source of truth. + +**Legend:** +- `[ ]` Module Name — not started +- `[ ]` *wip* Module Name — in progress +- `[x]` [Module Name](path/to/module/) — complete (linked) +- *skip* — skippable for fast-track learners + +--- + +## Study Path + +Phases are sequential but topics within a phase can be studied in parallel. +Prerequisites are noted — you can jump ahead if you already know the prereqs. + +| Phase | Focus | Topics | Prereqs | +|-------|-------|--------|---------| +| 1 | Toolkit & Literacy | Command Line, Git, Technical Writing, Programming Fundamentals, C, Python, Bash, Math | None | +| 2 | Core Theory | Computer Science, Electronics, Make & Dev Automation | Phase 1 | +| 3 | Systems & Design | Operating Systems, PCB Design, Web Tech, Project Management | Phase 2 | +| 4 | Security & Infra | Cyber Security, Ansible, Automation | Phases 2-3 | +| 5 | Embedded World | Embedded Systems, PCB Prototyping, Digital Fabrication | Phases 2-3 | +| 6 | Deep Systems | Kernel Development, Embedded OS/RTOS, IoT | Phases 3-5 | +| 7 | Intelligence & Integration | Computer Vision/Edge AI, Robotics, Rust, C++, Advanced Languages | Phases 5-6 | +| -- | Side Quest (anytime) | Global Geek Culture | None | + +**Fast-track:** If you have prior knowledge, check the prereqs list in each topic and skip what you know. Modules marked *skip* are optional/enrichment. + +--- + +## Phase 1 — Toolkit & Literacy + +### 1.1 Linux Command Line Tools +`path: linux-command-line/` +> Prereqs: None + +- [ ] Shell Basics — navigation, file ops, permissions, pipes, redirection, man pages +- [ ] File Management — find, locate, tar, rsync, ln, file, stat, du, df +- [ ] Text Processing — grep, sed, awk, cut, sort, uniq, tr, wc, jq, yq +- [ ] Process Management — ps, top/htop, kill, nice, nohup, jobs, bg/fg, screen/tmux +- [ ] Networking — ssh, curl, wget, netstat/ss, ip, dig, nmap, nc, scp/sftp +- [ ] System Administration — systemctl, journalctl, cron, users/groups, disk mgmt, mount +- [ ] Advanced Shell — parameter expansion, process substitution, subshells, traps, here-docs +- [ ] Power Tools — fzf, ripgrep, fd, bat, eza, delta, zoxide, direnv +- [ ] *skip* Customization — shell config, aliases, functions, prompt, dotfiles management +- [ ] Exercises & Projects — system monitor, backup script, log analyzer, file organizer + +### 1.2 Git +`path: git/` +> Prereqs: Command Line basics + +- [ ] Fundamentals — init, add, commit, status, log, diff, .gitignore +- [ ] Branching & Merging — branches, merge, fast-forward, merge conflicts +- [ ] Remote Collaboration — remotes, push, pull, fetch, clone, PRs, patches (email workflow) +- [ ] Advanced Operations — rebase, cherry-pick, bisect, reflog, stash, worktrees +- [ ] Internals — objects (blob/tree/commit/tag), refs, packfiles, how Git works +- [ ] Workflows & Strategies — feature branches, Git Flow, trunk-based, monorepo, submodules +- [ ] Hooks & Automation — pre-commit, commit-msg, CI integration, custom hooks +- [ ] *skip* Sourcehut-Specific — sr.ht workflows, mailing list patches, builds.sr.ht +- [ ] Exercises & Projects — contribute a patch, manage a multi-branch project, write custom hooks + +### 1.3 Technical Writing & Research Methodologies +`path: technical-writing/` +> Prereqs: None + +- [ ] Writing Fundamentals — clarity, structure, audience analysis, active voice, style guides +- [ ] Technical Documentation — API docs, user guides, READMEs, changelogs, man pages +- [ ] Research Methodology — literature review, hypothesis, experimental design, data collection +- [ ] Data Presentation — tables, charts, statistical visualization, LaTeX/typst basics +- [ ] *skip* Academic Writing — papers, citations, abstracts, peer review process +- [ ] Documentation Tools — Markdown, AsciiDoc, Sphinx, Doxygen, mdBook +- [ ] Exercises & Projects — write a project README, document an API, create a lab report + +### 1.4 Programming Fundamentals +`path: programming/fundamentals/` +> Prereqs: Command Line basics + +- [ ] Core Concepts — variables, types, expressions, control flow (all languages compared) +- [ ] Functions & Modularity — functions, scope, recursion, modules/packages +- [ ] Data Structures (language-level) — arrays, strings, hash maps, structs/classes +- [ ] I/O & File Handling — stdin/stdout, file read/write, serialization (JSON, TOML, CSV) +- [ ] Error Handling — exceptions, return codes, Result/Option types (cross-language comparison) +- [ ] Memory Models — stack vs heap, value vs reference, ownership overview +- [ ] Build & Run — compilers, interpreters, REPLs, build systems, package managers +- [ ] Testing Basics — unit tests, assertions, test frameworks per language +- [ ] Exercises & Projects — CLI calculator, file processor, multi-language Rosetta Stone set + +### 1.5 C +`path: programming/c/` +> Prereqs: Programming Fundamentals + +- [ ] C Basics — syntax, types, printf/scanf, compilation with gcc/clang +- [ ] Pointers & Memory — pointer arithmetic, malloc/free, arrays vs pointers, void pointers +- [ ] Strings & Arrays — C strings, buffer safety, multi-dimensional arrays +- [ ] Structs & Unions — struct layout, padding/alignment, bit fields, tagged unions +- [ ] Preprocessor — macros, conditional compilation, include guards, X-macros +- [ ] Standard Library — stdio, stdlib, string.h, math.h, time.h, errno +- [ ] File I/O & System Calls — fopen/fread, open/read/write, mmap, ioctl +- [ ] Build Systems — Makefiles for C, static/shared libraries, linking, pkg-config +- [ ] Debugging & Profiling — gdb, valgrind, sanitizers (ASan/UBSan), perf +- [ ] Advanced C — function pointers, callbacks, opaque types, C11/C17/C23 features +- [ ] Exercises & Projects — memory allocator, linked list library, simple shell, file parser + +### 1.6 Python +`path: programming/python/` +> Prereqs: Programming Fundamentals + +- [ ] Python Basics — syntax, types, REPL, scripts, pip, virtual environments (venv, uv) +- [ ] Data Structures — lists, tuples, dicts, sets, comprehensions, slicing, unpacking +- [ ] Functions & Modules — def, *args/**kwargs, lambda, imports, packages, \_\_name\_\_ +- [ ] OOP — classes, inheritance, dunder methods, properties, dataclasses, protocols +- [ ] Error Handling — exceptions, try/except/finally, custom exceptions, context managers +- [ ] File I/O & Data — file ops, pathlib, JSON, CSV, TOML, regex, struct/binary +- [ ] Standard Library — os, sys, subprocess, collections, itertools, functools, logging, argparse +- [ ] Testing — pytest, unittest, mocking, fixtures, parametrize, coverage +- [ ] Scientific Stack Intro — NumPy basics, Matplotlib, pandas intro, Jupyter notebooks +- [ ] Exercises & Projects — CLI tool, web scraper, data pipeline, test suite for a library + +### 1.7 Bash Scripting +`path: programming/bash/` +> Prereqs: Command Line Tools + +- [ ] Script Basics — shebang, variables, quoting, exit codes, set options (-euo pipefail) +- [ ] Control Flow — if/elif/else, case, for, while, until, select +- [ ] Functions — definition, local variables, return values, passing arguments +- [ ] Text Processing in Scripts — grep/sed/awk integration, regex, here-docs/here-strings +- [ ] Process & Job Control — subshells, command substitution, pipe chains, xargs, parallel +- [ ] Advanced Patterns — signal handling (trap), getopts, associative arrays, nameref +- [ ] Script Packaging — argument parsing, help messages, config files, logging, color output +- [ ] Exercises & Projects — deployment script, system health checker, interactive menu tool + +### 1.8 Mathematics Foundations +`path: mathematics/` +> Prereqs: None (high school math assumed) + +- [ ] Logic & Proofs — propositional logic, predicate logic, proof techniques, set theory +- [ ] Number Systems — integers, rationals, reals, complex numbers, modular arithmetic +- [ ] Linear Algebra — vectors, matrices, systems of equations, determinants, eigenvalues +- [ ] Calculus I — limits, derivatives, integrals, fundamental theorem +- [ ] Calculus II — techniques of integration, sequences, series, Taylor/Maclaurin +- [ ] Multivariable Calculus — partial derivatives, gradients, multiple integrals, vector fields +- [ ] Differential Equations — first/second order ODEs, Laplace transforms, systems of ODEs +- [ ] Probability & Statistics — combinatorics, probability theory, distributions, inference, Bayes +- [ ] Discrete Mathematics — graph theory, combinatorics, recurrences, number theory basics +- [ ] *skip* Numerical Methods — root finding, interpolation, numerical integration, optimization +- [ ] *skip* Transform Methods — Fourier series/transform, Z-transform, DFT/FFT +- [ ] Exercises & Projects — problem sets per module, computational exercises (Python/Julia) + +--- + +## Phase 2 — Core Theory + +### 2.1 Computer Science +`path: computer-science/` +> Prereqs: Programming Fundamentals, Math (discrete math, logic) + +- [ ] Data Structures — arrays, linked lists, stacks, queues, trees, heaps, graphs, hash tables +- [ ] Algorithms — sorting, searching, graph algorithms (BFS/DFS/Dijkstra), recursion +- [ ] Algorithm Design — divide & conquer, dynamic programming, greedy, backtracking +- [ ] Complexity Theory — Big-O/Omega/Theta, amortized analysis, P vs NP, reductions +- [ ] Computer Architecture — CPU design, instruction sets, pipelining, caches, memory hierarchy +- [ ] Networking — OSI model, TCP/IP, DNS, HTTP/HTTPS, sockets, routing +- [ ] Databases — relational model, SQL, normalization, indexing, transactions, NoSQL overview +- [ ] *skip* Compilers — lexing, parsing, ASTs, semantic analysis, code generation +- [ ] *skip* Distributed Systems — CAP theorem, consensus, replication, consistency models +- [ ] Exercises & Projects — implement core data structures (C + Python + Rust), solve algorithm sets + +### 2.2 Electronics +`path: electronics/` +> Prereqs: Math (calculus, linear algebra, complex numbers) + +- [ ] Circuit Fundamentals — voltage, current, resistance, power, Ohm's/Kirchhoff's laws +- [ ] DC Circuit Analysis — node/mesh analysis, Thevenin/Norton, superposition +- [ ] AC Circuits — phasors, impedance, resonance, frequency response, Bode plots +- [ ] Passive Components — resistors, capacitors, inductors, transformers, practical specs +- [ ] Diodes & Rectifiers — PN junction, diode types, rectifier circuits, voltage regulators +- [ ] Transistors (BJT) — operation, biasing, small-signal models, amplifier configurations +- [ ] Transistors (MOSFET) — operation, biasing, CMOS logic, switching applications +- [ ] Op-Amps — ideal model, inverting/non-inverting, integrator/differentiator, active filters +- [ ] Digital Electronics — logic gates, Boolean algebra, combinational circuits, sequential (flip-flops, counters) +- [ ] Sensors & Measurement — common sensors (temp, light, force, IMU), signal conditioning, ADC/DAC +- [ ] Power Electronics — linear regulators, switching converters (buck/boost), battery charging +- [ ] *skip* RF Basics — transmission lines, impedance matching, modulation, antenna fundamentals +- [ ] Exercises & Projects — voltage divider lab, amplifier design, logic circuit build, sensor interface + +### 2.3 Make & Dev Automation +`path: make-and-automation/` +> Prereqs: Command Line, Git, Programming (C or any compiled language) + +- [ ] Make Basics — targets, prerequisites, recipes, variables, automatic variables, phony targets +- [ ] Advanced Make — pattern rules, functions, conditional directives, auto-dependencies, recursive vs non-recursive +- [ ] CMake — CMakeLists.txt, targets, find_package, generator expressions, modern CMake idioms +- [ ] *skip* Meson — build definitions, dependencies, cross-compilation, wraps +- [ ] Task Runners — Justfile, Taskfile, npm scripts, Python invoke, cargo-make +- [ ] Dev Containers — Dockerfiles for dev, devcontainer spec, reproducible environments +- [ ] *skip* Documentation Generation — Doxygen, Sphinx autodoc, mdBook, man page generation +- [ ] Exercises & Projects — multi-target C Makefile, CMake project with libraries, polyglot build system + +--- + +## Phase 3 — Systems & Design + +### 3.1 Operating Systems +`path: operating-systems/` +> Prereqs: CS (architecture, data structures), C, Math (basic) + +- [ ] Introduction — OS role, history, types, system calls, kernel vs userspace +- [ ] Process Management — processes, threads, context switching, scheduling algorithms +- [ ] Synchronization — race conditions, mutexes, semaphores, deadlocks, condition variables +- [ ] Memory Management — virtual memory, paging, segmentation, page replacement, TLB +- [ ] File Systems — VFS, inodes, ext4, FAT, journaling, directory structures +- [ ] I/O & Devices — device drivers model, block vs character, DMA, interrupt handling +- [ ] IPC — pipes, shared memory, message queues, signals, sockets +- [ ] Networking in OS — socket API, TCP/IP stack, network drivers, netfilter +- [ ] Security & Protection — access control, capabilities, namespaces, cgroups, sandboxing +- [ ] *skip* Virtualization — hypervisors (type 1/2), containers, paravirtualization, KVM +- [ ] Exercises & Projects — system call tracer, simple scheduler simulation, xv6 labs + +### 3.2 PCB Design +`path: pcb-design/` +> Prereqs: Electronics (all fundamentals) + +- [ ] Fundamentals — schematic capture workflow, EDA tools overview (KiCad focus), libraries +- [ ] Component Selection — reading datasheets, footprints, sourcing, lifecycle management +- [ ] Schematic Design — hierarchical sheets, power symbols, net labels, design review checklist +- [ ] PCB Layout Basics — board outline, stackup, component placement strategy, routing basics +- [ ] Routing — trace width, via types, differential pairs, length matching, copper pours +- [ ] Power Distribution — PDN design, decoupling strategy, thermal management, power planes +- [ ] Signal Integrity — impedance control, crosstalk, EMI/EMC basics, ground planes +- [ ] *skip* High-Speed Design — controlled impedance, return paths, high-speed routing rules +- [ ] Design for Manufacturing — DFM/DFA rules, panelization, assembly drawings, BOM management +- [ ] Exercises & Projects — LED breakout board, sensor board (2-layer), power supply board (4-layer) + +### 3.3 Web Technologies +`path: web-technologies/` +> Prereqs: CS (networking, databases), Programming (Python or JS) + +- [ ] HTML & CSS — semantic HTML5, CSS layout (flexbox, grid), responsive design, accessibility +- [ ] JavaScript — ES6+, DOM manipulation, events, async/await, modules, fetch API +- [ ] Backend Fundamentals — HTTP protocol, REST API design, authentication/authorization +- [ ] Server-Side Development — Python (Flask/FastAPI), Node.js (Express), API development +- [ ] Databases for Web — SQL (PostgreSQL), ORM concepts, migrations, Redis caching +- [ ] *skip* Frontend Frameworks — component architecture, state management, SPA vs MPA concepts +- [ ] DevOps for Web — Docker, reverse proxies (nginx), TLS, CI/CD, deployment strategies +- [ ] Security — XSS, CSRF, CSP, SQL injection prevention, CORS, HTTPS +- [ ] Exercises & Projects — static site, REST API with database, full-stack CRUD app + +### 3.4 Project Management +`path: project-management/` +> Prereqs: Technical Writing + +- [ ] Fundamentals — project lifecycle, scope, stakeholders, constraints triangle +- [ ] Methodologies — Agile, Scrum, Kanban, Waterfall, hybrid approaches +- [ ] Planning — work breakdown structure, estimation, scheduling, Gantt charts, risk management +- [ ] Execution & Monitoring — KPIs, burndown charts, stand-ups, retrospectives +- [ ] Tools & Systems — issue trackers, Kanban boards, time tracking, documentation wikis +- [ ] *skip* Team Dynamics — communication, conflict resolution, remote collaboration +- [ ] Engineering-Specific — hardware project management, firmware release cycles, versioning +- [ ] Exercises & Projects — plan a personal project end-to-end, set up a Kanban workflow + +--- + +## Phase 4 — Security & Infrastructure + +### 4.1 Cyber Security +`path: cyber-security/` +> Prereqs: CS (networking, OS concepts), Command Line, Programming + +- [ ] Fundamentals — CIA triad, threat modeling, attack surfaces, defense in depth +- [ ] Network Security — firewalls, IDS/IPS, VPN, packet analysis, network scanning +- [ ] System Hardening — access control, SELinux/AppArmor, audit logging, patching, CIS benchmarks +- [ ] Cryptography — symmetric/asymmetric encryption, hashing, TLS/SSL, PKI, key management +- [ ] Application Security — OWASP top 10, input validation, secure coding practices, code auditing +- [ ] Embedded Security — secure boot, firmware encryption, hardware attacks, side-channel basics +- [ ] *skip* Penetration Testing — methodology, recon, exploitation, post-exploitation, reporting +- [ ] *skip* Incident Response — detection, containment, forensics, recovery, lessons learned +- [ ] Exercises & Projects — harden a Linux server, set up TLS, CTF challenges, vulnerability assessment + +### 4.2 Ansible +`path: ansible/` +> Prereqs: Command Line (advanced), SSH, Linux administration + +- [ ] Introduction — IaC concepts, Ansible architecture, installation, configuration +- [ ] Core Concepts — inventory, modules, ad-hoc commands, YAML syntax +- [ ] Playbooks — plays, tasks, handlers, variables, facts, conditionals, loops +- [ ] Roles & Organization — role structure, Galaxy, collections, reuse patterns, project layout +- [ ] Advanced — Jinja2 templates, filters, lookups, custom modules, vault (secrets) +- [ ] *skip* Cloud & Containers — cloud provisioning, Docker management, Kubernetes basics +- [ ] Testing — Molecule, ansible-lint, CI integration, idempotency testing +- [ ] Exercises & Projects — server setup playbook, multi-tier deployment, homelab automation + +### 4.3 Automation +`path: automation/` +> Prereqs: Command Line, Bash, Git, Make basics + +- [ ] Principles — DRY, idempotency, reproducibility, when to automate (ROI) +- [ ] Shell Automation — cron, systemd timers, inotify/file watching, at +- [ ] CI/CD — concepts, GitHub Actions, sourcehut builds, pipeline design, artifacts +- [ ] Testing Automation — unit/integration/e2e strategies, fuzzing, property-based testing +- [ ] Infrastructure Automation — Terraform concepts, Ansible integration, GitOps basics +- [ ] *skip* Monitoring & Alerting — Prometheus, Grafana, alerting rules, log aggregation +- [ ] Workflow Automation — pre-commit hooks, linters, formatters, changelog generation +- [ ] Exercises & Projects — CI pipeline for a project, pre-commit setup, automated backup system + +--- + +## Phase 5 — Embedded World + +### 5.1 Embedded Systems +`path: embedded-systems/` +> Prereqs: Electronics, C, OS concepts + +- [ ] Introduction — embedded vs general-purpose, architectures (ARM Cortex-M, RISC-V, AVR), dev boards +- [ ] Microcontroller Fundamentals — CPU architecture, memory map, registers, clock system, startup code +- [ ] GPIO & Basic I/O — digital I/O, pull-ups/downs, debouncing, LED/button interfacing +- [ ] Interrupts — interrupt vectors, ISRs, priority, NVIC (ARM), critical sections +- [ ] Timers & PWM — timer modes, input capture, output compare, PWM generation, frequency measurement +- [ ] Serial Communication — UART (theory + implementation), SPI, I2C, protocol debugging +- [ ] ADC & DAC — conversion principles, sampling, resolution, DMA-driven ADC, signal conditioning +- [ ] DMA — DMA controller, channels, circular mode, memory-to-peripheral, performance gains +- [ ] Memory — flash programming, linker scripts, bootloader concepts, EEPROM, external memory +- [ ] Low Power Design — sleep modes, wakeup sources, power budgets, current measurement +- [ ] Communication Protocols — CAN bus, Modbus, 1-Wire, wireless overview (BLE, WiFi, LoRa) +- [ ] HAL vs Bare-Metal — vendor HALs (STM32 HAL/LL), CMSIS, register-level programming +- [ ] Debugging & Tools — JTAG/SWD, OpenOCD, GDB for embedded, logic analyzer, oscilloscope +- [ ] Exercises & Projects — blinky to bare-metal, UART logger, SPI sensor driver, motor controller + +### 5.2 PCB Prototyping +`path: pcb-prototyping/` +> Prereqs: PCB Design, Electronics + +- [ ] Breadboarding & Perfboard — prototyping techniques, Manhattan style, wire wrapping +- [ ] PCB Milling — desktop CNC (3018, Bantam), FlatCAM, isolation routing, drill files +- [ ] Chemical Etching — toner transfer, UV photoresist, etchant chemistry, safety +- [ ] Professional Fabrication — Gerber/drill generation, fab house selection (JLCPCB, PCBWay), ordering +- [ ] Assembly — hand soldering (through-hole + SMD), reflow (hot plate, oven), stencils +- [ ] Testing & Debugging — continuity testing, power-on sequence, functional testing, rework +- [ ] *skip* Rapid Iteration — version control for hardware, ECOs, change management +- [ ] Exercises & Projects — etch a simple board, order and assemble a PCB, full prototype cycle + +### 5.3 Digital Fabrication +`path: digital-fabrication/` +> Prereqs: Electronics basics, CAD basics + +- [ ] Overview — fab labs, maker culture, digital fabrication workflow +- [ ] 3D Printing — FDM, SLA/DLP, materials, slicing (Cura/PrusaSlicer), design for 3D printing +- [ ] Laser Cutting — materials, kerf compensation, design for laser, power/speed settings, safety +- [ ] *skip* CNC Machining — milling basics, G-code, CAM (FreeCAD Path), materials, work-holding +- [ ] CAD for Fabrication — FreeCAD, OpenSCAD, parametric design, export formats (STL, DXF, STEP) +- [ ] Enclosure Design — designing for electronics, standoffs, cable routing, thermal considerations +- [ ] *skip* Mold Making & Casting — silicone molds, resin casting, urethane, finishing +- [ ] Exercises & Projects — 3D print an enclosure, laser-cut a panel, design a complete housing + +--- + +## Phase 6 — Deep Systems + +### 6.1 Kernel Development +`path: kernel-development/` +> Prereqs: OS, C (advanced), Computer Architecture + +- [ ] Introduction — kernel role, monolithic vs micro, Linux source tree navigation +- [ ] Build & Boot — kernel configuration, cross-compiling, QEMU/VM testing, boot process +- [ ] Kernel Modules — module structure, Makefiles, loading/unloading, module parameters, licensing +- [ ] Character Devices — file_operations, major/minor numbers, ioctl, sysfs, procfs +- [ ] Kernel Memory — kmalloc/kfree, vmalloc, slab allocator, page allocator, GFP flags +- [ ] Concurrency — spinlocks, mutexes, RCU, atomic operations, per-CPU data, completion +- [ ] Interrupts & Timers — top/bottom halves, tasklets, workqueues, hrtimers, softirqs +- [ ] Device Model — platform devices, device tree, driver model, buses, probe/remove +- [ ] *skip* Networking — net_device, socket buffers (skb), netfilter hooks, packet flow +- [ ] *skip* File Systems — VFS interface, implementing a simple filesystem, block layer +- [ ] Debugging — printk/dynamic debug, ftrace, kprobes, KGDB, crash dump analysis, KASAN +- [ ] Exercises & Projects — hello-world module, char device driver, device tree overlay, simple driver + +### 6.2 Embedded OS & RTOS +`path: embedded-os/` +> Prereqs: Embedded Systems, OS, C + +- [ ] RTOS Concepts — real-time constraints, hard/soft RT, scheduling (rate monotonic, EDF), WCET +- [ ] FreeRTOS — tasks, queues, semaphores/mutexes, timers, memory management, portability +- [ ] Zephyr — architecture, devicetree, Kconfig, kernel primitives, networking stack, west tool +- [ ] Embedded Linux — Buildroot, Yocto/OpenEmbedded, cross-toolchains, root filesystem, device tree +- [ ] Bootloaders — U-Boot, MCUboot, boot stages, secure boot chain +- [ ] Bare-Metal vs RTOS vs Linux — decision framework, resource constraints, trade-offs +- [ ] Inter-Task Communication — message queues, event groups, mailboxes, shared memory, priority inversion +- [ ] *skip* OTA Updates — firmware update strategies, A/B partitions, rollback, delta updates +- [ ] Exercises & Projects — FreeRTOS multitask app, Zephyr sensor driver, Buildroot custom image + +### 6.3 Internet of Things +`path: iot/` +> Prereqs: Embedded Systems, Web Tech (backend), Networking + +- [ ] IoT Architecture — edge/fog/cloud, reference architectures, design patterns +- [ ] Connectivity — WiFi, BLE, LoRa/LoRaWAN, Zigbee, Thread, protocol comparison +- [ ] Messaging Protocols — MQTT (deep dive), CoAP, HTTP for IoT, WebSockets +- [ ] IoT Platforms — Home Assistant, Node-RED, Thingsboard, AWS IoT Core overview +- [ ] Data Pipeline — collection, time-series databases (InfluxDB), processing, dashboards (Grafana) +- [ ] Edge Computing — local processing, gateway design, edge inference, offline operation +- [ ] Security & Privacy — device identity, secure communication, firmware signing, threat landscape +- [ ] *skip* Standards — Matter, Thread, OPC-UA, industrial IoT +- [ ] Exercises & Projects — MQTT sensor network, LoRa node + gateway, home automation system + +--- + +## Phase 7 — Intelligence & Integration + +### 7.1 Computer Vision & Edge AI +`path: computer-vision/` +> Prereqs: Math (linear algebra, calculus, probability), Python, Embedded basics + +- [ ] Image Fundamentals — digital images, color spaces, histograms, geometric transforms +- [ ] Image Processing — convolution, filtering, morphology, edge detection, thresholding +- [ ] Feature Detection — corners (Harris, Shi-Tomasi), descriptors (SIFT/ORB), feature matching +- [ ] Classical CV — template matching, object detection (Haar/HOG), tracking, optical flow +- [ ] Deep Learning Foundations — neural networks, backprop, CNNs, training loop, PyTorch/TensorFlow +- [ ] Object Detection & Segmentation — YOLO, SSD, U-Net, Mask R-CNN, transfer learning +- [ ] Model Optimization — quantization, pruning, knowledge distillation, ONNX, TFLite +- [ ] Edge Deployment — Coral TPU, Jetson, OpenMV, RPi, NPU/VPU, inference frameworks +- [ ] *skip* Advanced Topics — 3D vision, depth estimation, visual SLAM, generative models +- [ ] Exercises & Projects — OpenCV pipeline, train a classifier, deploy on edge device, visual inspection + +### 7.2 Robotics +`path: robotics/` +> Prereqs: Math (linear algebra, calculus, diff eq), Electronics, Embedded, Programming (C++/Python) + +- [ ] Foundations — robot types, coordinate frames, homogeneous transforms, DH convention +- [ ] Kinematics — forward/inverse kinematics, Jacobian, velocity kinematics, workspace analysis +- [ ] Dynamics — Newton-Euler, Lagrangian mechanics, torque computation, gravity compensation +- [ ] Actuators & Sensors — DC/stepper/servo motors, encoders, IMUs, LIDAR, depth cameras +- [ ] Control Systems — PID control, state-space, trajectory generation, motion profiles +- [ ] Motion Planning — path planning (A*, RRT, PRM), obstacle avoidance, configuration space +- [ ] *skip* SLAM & Localization — EKF/particle filter, lidar SLAM, visual SLAM, sensor fusion +- [ ] ROS 2 — architecture, nodes, topics, services, actions, launch files, URDF, Gazebo +- [ ] *skip* Mobile Robots — wheeled (differential, omnidirectional), legged basics, drone basics +- [ ] Exercises & Projects — PID line follower, robotic arm control, ROS 2 navigation stack + +### 7.3 Rust +`path: programming/rust/` +> Prereqs: Programming Fundamentals, C (recommended) + +- [ ] Getting Started — cargo, crate structure, hello world, rustup, editions +- [ ] Ownership & Borrowing — ownership rules, references, lifetimes, borrow checker +- [ ] Type System — structs, enums, pattern matching, generics, traits, trait objects +- [ ] Error Handling — Result, Option, ? operator, custom errors, anyhow/thiserror +- [ ] Collections & Iterators — Vec, HashMap, iterator adaptors, closures, functional patterns +- [ ] Concurrency — threads, channels, Arc/Mutex, Send/Sync, async/await, tokio basics +- [ ] Unsafe & FFI — unsafe blocks, raw pointers, extern "C", bindgen, calling C from Rust +- [ ] Ecosystem — serde, clap, reqwest, embedded Rust (no_std), testing, benchmarking +- [ ] Exercises & Projects — CLI tool, concurrent web scraper, embedded Rust blinky, C FFI wrapper + +### 7.4 C++ +`path: programming/cpp/` +> Prereqs: C, Programming Fundamentals + +- [ ] C++ Basics — classes, constructors/destructors, namespaces, references, streams +- [ ] OOP — inheritance, polymorphism, virtual functions, abstract classes, RTTI +- [ ] Templates — function/class templates, template specialization, variadic templates, SFINAE basics +- [ ] STL — containers, iterators, algorithms, string, functional +- [ ] Modern C++ (17/20/23) — auto, structured bindings, ranges, concepts, modules, coroutines overview +- [ ] Memory Management — RAII, smart pointers (unique/shared/weak), move semantics, rule of 5 +- [ ] Concurrency — std::thread, mutex, condition_variable, atomic, futures, async +- [ ] *skip* Advanced — type erasure, CRTP, policy-based design, compile-time computation +- [ ] Exercises & Projects — matrix library, concurrent data structure, embedded C++ on MCU + +### 7.5 Advanced Languages +`path: programming/advanced-languages/` +> Prereqs: Programming Fundamentals, specific domain knowledge per language + +- [ ] Python Advanced — decorators, generators, metaclasses, asyncio, C extensions, type hints +- [ ] Lua — tables, metatables, coroutines, embedding in C, LuaJIT FFI, game scripting +- [ ] Julia — multiple dispatch, type system, macros, performance, scientific computing, GPU +- [ ] R — data frames, tidyverse, ggplot2, statistical modeling, R Markdown +- [ ] Cross-Language — FFI patterns, polyglot projects, language interop, choosing the right tool +- [ ] Exercises & Projects — Python C extension, Lua-embedded app, Julia scientific notebook, R analysis + +--- + +## Side Quest — Anytime + +### S.1 Global Geek Culture +`path: geek-culture/` +> Prereqs: None — read alongside any phase + +- [ ] Hacker Culture Origins — MIT AI Lab, Tech Model Railroad Club, PDP-1, hacker ethic, "Hackers" (Levy) +- [ ] Phone Phreaking & Early Hacking — Captain Crunch, blue boxes, 2600 magazine, WarGames, Legion of Doom +- [ ] The Jargon File & Hacker Lore — Jargon File, INTERCAL, RFC humor, koans, AI koans, quines, IOCCC +- [ ] Text File Underground — textfiles.com, e-zines (Phrack, LoD/H Technical Journal), BBS culture, ANSI art +- [ ] The Geek Code & .sig Culture — Geek Code blocks, ASCII art sigs, fortune files, Plan files, finger protocol +- [ ] Usenet & Early Internet Culture — flame wars, FAQs, killfiles, alt.* hierarchy, trolling as art, Eternal September +- [ ] Cryptography Wars & Cypherpunks — cypherpunk mailing list, PGP, Clipper chip, Zimmermann, remailers +- [ ] Demoscene — demo groups, size coding (4k/64k intros), Amiga/Atari scene, Revision/Assembly parties, pouet.net +- [ ] Open Source Movement — GNU Manifesto, FSF, Cathedral vs Bazaar, Linux, copyleft vs permissive, forking culture +- [ ] Warez & Underground Scenes — warez groups, NFOs, cracktros, courier culture, The Scene +- [ ] Internet Mysteries & Challenges — Cicada 3301, Webdriver Torso, Markovian Parallax Denigrate, crypto puzzles, ARGs +- [ ] MUDs, MOOs & Virtual Worlds — text adventures, LambdaMOO, online community origins, digital identity +- [ ] Hacker Conferences & Community — DEF CON, CCC, 2600 meetings, hackerspaces, CTF culture +- [ ] Maker & Hardware Hacking — hackerspaces, CCC hardware, Arduino/RPi, OSHW, right to repair, modding +- [ ] *skip* Cyberpunk & Sci-Fi Canon — Gibson, Stephenson, Dick, Doctorow, technology predictions +- [ ] *skip* Global Perspectives — CCC vs DEF CON culture, Shenzhen, Global South tech, internet freedom + +--- + +## Summary + +| # | Topic | Modules | Status | +|---|-------|---------|--------| +| 1.1 | Linux Command Line | 10 | Not started | +| 1.2 | Git | 9 | Not started | +| 1.3 | Technical Writing | 7 | Not started | +| 1.4 | Programming Fundamentals | 9 | Not started | +| 1.5 | C | 11 | Not started | +| 1.6 | Python | 10 | Not started | +| 1.7 | Bash Scripting | 8 | Not started | +| 1.8 | Mathematics | 12 | Not started | +| 2.1 | Computer Science | 10 | Not started | +| 2.2 | Electronics | 13 | Not started | +| 2.3 | Make & Dev Automation | 8 | Not started | +| 3.1 | Operating Systems | 11 | Not started | +| 3.2 | PCB Design | 10 | Not started | +| 3.3 | Web Technologies | 9 | Not started | +| 3.4 | Project Management | 8 | Not started | +| 4.1 | Cyber Security | 9 | Not started | +| 4.2 | Ansible | 8 | Not started | +| 4.3 | Automation | 8 | Not started | +| 5.1 | Embedded Systems | 14 | Not started | +| 5.2 | PCB Prototyping | 8 | Not started | +| 5.3 | Digital Fabrication | 8 | Not started | +| 6.1 | Kernel Development | 12 | Not started | +| 6.2 | Embedded OS & RTOS | 9 | Not started | +| 6.3 | IoT | 9 | Not started | +| 7.1 | Computer Vision & Edge AI | 10 | Not started | +| 7.2 | Robotics | 10 | Not started | +| 7.3 | Rust | 9 | Not started | +| 7.4 | C++ | 9 | Not started | +| 7.5 | Advanced Languages | 6 | Not started | +| S.1 | Geek Culture | 16 | Not started | diff --git a/README.md b/README.md new file mode 100644 index 0000000..a25b9ca --- /dev/null +++ b/README.md @@ -0,0 +1,38 @@ +# ubergeek + +A comprehensive, self-paced study guide covering electronics, embedded +systems, programming, computer science, cybersecurity, and more. Built as a +sourcehut wiki. + +## Topics + +30 topics across 7 phases — from command line basics to robotics and edge AI. +See [PROGRESS.md](PROGRESS.md) for the full topic list and study roadmap. + +## Structure + +Each topic follows the same layout: + +``` +topic-name/ +├── index.md # overview and module list +├── images/ # diagrams, schematics, screenshots +├── samples/ # runnable code and project files +└── module-name/ + └── index.md # module content +``` + +## Links + +- **Sourcehut (primary):** https://git.sr.ht/~gumxcc/ubergeek +- **Wiki:** https://man.sr.ht/~gumxcc/ubergeek/ + +## Contributing + +See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on submitting patches, +pull requests, or merge requests. + +## License + +Content is [CC BY-SA 4.0](LICENSES/CC-BY-SA-4.0). Code samples are +[MIT](LICENSES/MIT). See [LICENSE.md](LICENSE.md) for details. diff --git a/build.sh b/build.sh new file mode 100644 index 0000000..2973c2f --- /dev/null +++ b/build.sh @@ -0,0 +1,215 @@ +#!/bin/bash +# build.sh - ubergeek static site generator +# Converts the wiki markdown structure into HTML using lowdown + mo. + +[[ "$(basename "${0}")" == "build.sh" ]] || { echo "Do not source build.sh"; exit 1; } + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [[ ! -f "${SCRIPT_DIR}/build/mo" ]]; then + echo "[error] build/mo not found. Run: curl -sL https://raw.githubusercontent.com/tests-always-included/mo/master/mo -o build/mo && chmod +x build/mo" >&2 + exit 1 +fi +. "${SCRIPT_DIR}/build/mo" + +FORCE=false +while [[ $# -gt 0 ]]; do + case "$1" in + -f|--force) FORCE=true ;; + *) echo "[error] Unknown option: $1" >&2; exit 1 ;; + esac + shift +done + +TMPDIR_BUILD="$(mktemp -d)" +cleanup() { rm -rf "${TMPDIR_BUILD}"; } +trap cleanup EXIT INT TERM + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +get_frontmatter() { + local key="$1" file="$2" + lowdown -X "${key}" "${file}" 2>/dev/null || true +} + +check_has_code() { + local html="$1" + if echo "${html}" | grep -qE '<pre><code|<code class='; then + HAS_CODE="true" + else + HAS_CODE="" + fi +} + +render_page() { + local template="$1" outfile="$2" + mkdir -p "$(dirname "${outfile}")" + local tname="${template##build/templates/}" + cp "${template}" "build/templates/partials/${tname}" + (cd build/templates/partials && mo "${tname}") > "${outfile}" + rm -f "build/templates/partials/${tname}" +} + +build_breadcrumbs() { + local path="$1" + BREADCRUMBS="" + local components="${path}" + while [[ "${components}" != "." ]]; do + BREADCRUMBS=" / <a href=\"/${components}\">$(basename "${components}")</a>${BREADCRUMBS}" + components="$(dirname "${components}")" + done +} + +# --------------------------------------------------------------------------- +# process_dir dir +# Recursively renders topic and module pages, returns a markdown list +# of direct children into ${TMPDIR_BUILD}/${dir//\//_}.list.md +# --------------------------------------------------------------------------- +process_dir() { + local dir="$1" + local list_file="${TMPDIR_BUILD}/${dir//\//_}.list.md" + : > "${list_file}" + + local find_list="${TMPDIR_BUILD}/${dir//\//_}.find.list" + find "${dir}" -mindepth 1 -maxdepth 1 -not -name '.*' -print0 | sort -z > "${find_list}" + + while IFS= read -r -d '' node; do + if [[ -d "${node}" ]]; then + local subdir_title + subdir_title="$(get_frontmatter title "${node}/index.md" 2>/dev/null || true)" + [[ -z "${subdir_title}" ]] && subdir_title="$(basename "${node}")" + printf -- '- [%s](/%s)\n' "${subdir_title}" "${node}" >> "${list_file}" + process_dir "${node}" + + elif [[ "${node}" == *.md && "$(basename "${node}")" != "index.md" ]]; then + local slug="${node%.md}" + local title date lang + title="$(get_frontmatter title "${node}")" + [[ -z "${title}" ]] && title="$(basename "${slug}")" + date="$(get_frontmatter date "${node}")" + lang="$(get_frontmatter lang "${node}")" + + local out_html="output/${slug}/index.html" + local content + content="$(lowdown "${node}")" + check_has_code "${content}" + build_breadcrumbs "${slug}" + + export PAGE_TITLE="${title}" + export CONTENT="${content}" + export BREADCRUMBS + export HEADER_EXTRA="${date}" + export META_DATE="${date}" + export META_DESCRIPTION="" + export HAS_CODE + export LANG="${lang:-en}" + export DIR="" + + render_page "build/templates/page.html" "${out_html}" + printf -- '- [%s](/%s)\n' "${title}" "${slug}" >> "${list_file}" + echo "[built] ${out_html}" + fi + done < "${find_list}" + + # Build this directory's index page + local index_src="${dir}/index.md" + local index_md="${TMPDIR_BUILD}/${dir//\//_}.index.md" + local dir_title + + if [[ -f "${index_src}" ]]; then + dir_title="$(get_frontmatter title "${index_src}")" + cp "${index_src}" "${index_md}" + else + dir_title="$(basename "${dir}")" + printf '# %s\n\n' "${dir_title}" > "${index_md}" + fi + printf '\n\n' >> "${index_md}" + cat "${list_file}" >> "${index_md}" + + local out_html="output/${dir}/index.html" + local content + content="$(lowdown "${index_md}")" + check_has_code "${content}" + build_breadcrumbs "${dir}" + + export PAGE_TITLE="${dir_title}" + export CONTENT="${content}" + export BREADCRUMBS + export HEADER_EXTRA="" + export META_DATE="" + export META_DESCRIPTION="" + export HAS_CODE + export LANG="en" + export DIR="" + + render_page "build/templates/page.html" "${out_html}" + echo "[built] output/${dir}/index.html" +} + +# --------------------------------------------------------------------------- +# Main build +# --------------------------------------------------------------------------- + +mkdir -p output + +# Discover top-level topic directories (exclude hidden, build, output) +topics_list="${TMPDIR_BUILD}/topics.list.md" +: > "${topics_list}" + +find_list="${TMPDIR_BUILD}/toplevel.find.list" +find . -mindepth 1 -maxdepth 1 -type d \ + -not -name '.*' -not -name 'build' -not -name 'output' \ + -print0 | sort -z > "${find_list}" + +while IFS= read -r -d '' topic; do + topic="${topic#./}" + topic_title="$(get_frontmatter title "${topic}/index.md" 2>/dev/null || true)" + [[ -z "${topic_title}" ]] && topic_title="${topic}" + printf -- '- [%s](/%s)\n' "${topic_title}" "${topic}" >> "${topics_list}" + process_dir "${topic}" +done < "${find_list}" + +# Home page — README.md + topic listing +home_md="${TMPDIR_BUILD}/home.md" +cp README.md "${home_md}" +printf '\n\n## topics:\n\n' >> "${home_md}" +cat "${topics_list}" >> "${home_md}" + +content="$(lowdown "${home_md}")" +check_has_code "${content}" + +export PAGE_TITLE="" +export CONTENT="${content}" +export BREADCRUMBS="" +export HEADER_EXTRA="" +export META_DATE="" +export META_DESCRIPTION="" +export HAS_CODE +export LANG="en" +export DIR="" + +render_page "build/templates/home.html" "output/index.html" +echo "[built] output/index.html" + +# License page +if [[ -f "LICENSE.md" ]]; then + content="$(lowdown LICENSE.md)" + check_has_code "${content}" + export PAGE_TITLE="license" + export CONTENT="${content}" + export BREADCRUMBS=" / <a href=\"/license\">license</a>" + export HEADER_EXTRA="" + export META_DATE="" + export META_DESCRIPTION="" + export HAS_CODE + export LANG="en" + export DIR="" + render_page "build/templates/page.html" "output/license/index.html" + echo "[built] output/license/index.html" +fi + +echo "" +echo "[done] Build complete. Output in output/" diff --git a/build/mo b/build/mo new file mode 100755 index 0000000..8247a2d --- /dev/null +++ b/build/mo @@ -0,0 +1,2037 @@ +#!/usr/bin/env bash +# +#/ Mo is a mustache template rendering software written in bash. It inserts +#/ environment variables into templates. +#/ +#/ Simply put, mo will change {{VARIABLE}} into the value of that +#/ environment variable. You can use {{#VARIABLE}}content{{/VARIABLE}} to +#/ conditionally display content or iterate over the values of an array. +#/ +#/ Learn more about mustache templates at https://mustache.github.io/ +#/ +#/ Simple usage: +#/ +#/ mo [OPTIONS] filenames... +#/ +#/ Options: +#/ +#/ --allow-function-arguments +#/ Permit functions to be called with additional arguments. Otherwise, +#/ the only way to get access to the arguments is to use the +#/ MO_FUNCTION_ARGS environment variable. +#/ -d, --debug +#/ Enable debug logging to stderr. +#/ -u, --fail-not-set +#/ Fail upon expansion of an unset variable. Will silently ignore by +#/ default. Alternately, set MO_FAIL_ON_UNSET to a non-empty value. +#/ -x, --fail-on-function +#/ Fail when a function returns a non-zero status code instead of +#/ silently ignoring it. Alternately, set MO_FAIL_ON_FUNCTION to a +#/ non-empty value. +#/ -f, --fail-on-file +#/ Fail when a file (from command-line or partial) does not exist. +#/ Alternately, set MO_FAIL_ON_FILE to a non-empty value. +#/ -e, --false +#/ Treat the string "false" as empty for conditionals. Alternately, +#/ set MO_FALSE_IS_EMPTY to a non-empty value. +#/ -h, --help +#/ This message. +#/ -s=FILE, --source=FILE +#/ Load FILE into the environment before processing templates. +#/ Can be used multiple times. The file must be a valid shell script +#/ and should only contain variable assignments. +#/ -o=DELIM, --open=DELIM +#/ Set the opening delimiter. Default is "{{". +#/ -c=DELIM, --close=DELIM +#/ Set the closing delimiter. Default is "}}". +#/ -- Indicate the end of options. All arguments after this will be +#/ treated as filenames only. Use when filenames may start with +#/ hyphens. +#/ +#/ Mo uses the following environment variables: +#/ +#/ MO_ALLOW_FUNCTION_ARGUMENTS - When set to a non-empty value, this allows +#/ functions referenced in templates to receive additional options and +#/ arguments. +#/ MO_CLOSE_DELIMITER - The string used when closing a tag. Defaults to "}}". +#/ Used internally. +#/ MO_CLOSE_DELIMITER_DEFAULT - The default value of MO_CLOSE_DELIMITER. Used +#/ when resetting the close delimiter, such as when parsing a partial. +#/ MO_CURRENT - Variable name to use for ".". +#/ MO_DEBUG - When set to a non-empty value, additional debug information is +#/ written to stderr. +#/ MO_FUNCTION_ARGS - Arguments passed to the function. +#/ MO_FAIL_ON_FILE - If a filename from the command-line is missing or a +#/ partial does not exist, abort with an error. +#/ MO_FAIL_ON_FUNCTION - If a function returns a non-zero status code, abort +#/ with an error. +#/ MO_FAIL_ON_UNSET - When set to a non-empty value, expansion of an unset env +#/ variable will be aborted with an error. +#/ MO_FALSE_IS_EMPTY - When set to a non-empty value, the string "false" will +#/ be treated as an empty value for the purposes of conditionals. +#/ MO_OPEN_DELIMITER - The string used when opening a tag. Defaults to "{{". +#/ Used internally. +#/ MO_OPEN_DELIMITER_DEFAULT - The default value of MO_OPEN_DELIMITER. Used +#/ when resetting the open delimiter, such as when parsing a partial. +#/ MO_ORIGINAL_COMMAND - Used to find the `mo` program in order to generate a +#/ help message. +#/ MO_PARSED - Content that has made it through the template engine. +#/ MO_STANDALONE_CONTENT - The unparsed content that preceeded the current tag. +#/ When a standalone tag is encountered, this is checked to see if it only +#/ contains whitespace. If this and the whitespace condition after a tag is +#/ met, then this will be reset to $'\n'. +#/ MO_UNPARSED - Template content yet to make it through the parser. +#/ +#/ Mo is under a MIT style licence with an additional non-advertising clause. +#/ See LICENSE.md for the full text. +#/ +#/ This is open source! Please feel free to contribute. +#/ +#/ https://github.com/tests-always-included/mo + +#: Disable these warnings for the entire file +#: +#: VAR_NAME was modified in a subshell. That change might be lost. +# shellcheck disable=SC2031 +#: +#: Modification of VAR_NAME is local (to subshell caused by (..) group). +# shellcheck disable=SC2030 + +# Public: Template parser function. Writes templates to stdout. +# +# $0 - Name of the mo file, used for getting the help message. +# $@ - Filenames to parse. +# +# Returns nothing. +mo() ( + local moSource moFiles moDoubleHyphens moParsed moContent + + #: This function executes in a subshell; IFS is reset at the end. + IFS=$' \n\t' + + #: Enable a strict mode. This is also reset at the end. + set -eEu -o pipefail + moFiles=() + moDoubleHyphens=false + MO_OPEN_DELIMITER_DEFAULT="{{" + MO_CLOSE_DELIMITER_DEFAULT="}}" + MO_FUNCTION_CACHE_HIT=() + MO_FUNCTION_CACHE_MISS=() + + if [[ $# -gt 0 ]]; then + for arg in "$@"; do + if $moDoubleHyphens; then + #: After we encounter two hyphens together, all the rest + #: of the arguments are files. + moFiles=(${moFiles[@]+"${moFiles[@]}"} "$arg") + else + case "$arg" in + -h|--h|--he|--hel|--help|-\?) + mo::usage "$0" + exit 0 + ;; + + --allow-function-arguments) + MO_ALLOW_FUNCTION_ARGUMENTS=true + ;; + + -u | --fail-not-set) + MO_FAIL_ON_UNSET=true + ;; + + -x | --fail-on-function) + MO_FAIL_ON_FUNCTION=true + ;; + + -p | --fail-on-file) + MO_FAIL_ON_FILE=true + ;; + + -e | --false) + MO_FALSE_IS_EMPTY=true + ;; + + -s=* | --source=*) + if [[ "$arg" == --source=* ]]; then + moSource="${arg#--source=}" + else + moSource="${arg#-s=}" + fi + + if [[ -e "$moSource" ]]; then + # shellcheck disable=SC1090 + . "$moSource" + else + echo "No such file: $moSource" >&2 + exit 1 + fi + ;; + + -o=* | --open=*) + if [[ "$arg" == --open=* ]]; then + MO_OPEN_DELIMITER_DEFAULT="${arg#--open=}" + else + MO_OPEN_DELIMITER_DEFAULT="${arg#-o=}" + fi + ;; + + -c=* | --close=*) + if [[ "$arg" == --close=* ]]; then + MO_CLOSE_DELIMITER_DEFAULT="${arg#--close=}" + else + MO_CLOSE_DELIMITER_DEFAULT="${arg#-c=}" + fi + ;; + + -d | --debug) + MO_DEBUG=true + ;; + + --) + #: Set a flag indicating we've encountered double hyphens + moDoubleHyphens=true + ;; + + -*) + mo::error "Unknown option: $arg (See --help for options)" + ;; + + *) + #: Every arg that is not a flag or a option should be a file + moFiles=(${moFiles[@]+"${moFiles[@]}"} "$arg") + ;; + esac + fi + done + fi + + mo::debug "Debug enabled" + MO_OPEN_DELIMITER="$MO_OPEN_DELIMITER_DEFAULT" + MO_CLOSE_DELIMITER="$MO_CLOSE_DELIMITER_DEFAULT" + mo::content moContent ${moFiles[@]+"${moFiles[@]}"} || return 1 + mo::parse moParsed "$moContent" + echo -n "$moParsed" +) + + +# Internal: Show a debug message +# +# $1 - The debug message to show +# +# Returns nothing. +mo::debug() { + if [[ -n "${MO_DEBUG:-}" ]]; then + echo "DEBUG ${FUNCNAME[1]:-?} - $1" >&2 + fi +} + + +# Internal: Show a debug message and internal state information +# +# No arguments +# +# Returns nothing. +mo::debugShowState() { + if [[ -z "${MO_DEBUG:-}" ]]; then + return + fi + + local moState moTemp moIndex moDots + + mo::escape moTemp "$MO_OPEN_DELIMITER" + moState="open: $moTemp" + mo::escape moTemp "$MO_CLOSE_DELIMITER" + moState="$moState close: $moTemp" + mo::escape moTemp "$MO_STANDALONE_CONTENT" + moState="$moState standalone: $moTemp" + mo::escape moTemp "$MO_CURRENT" + moState="$moState current: $moTemp" + moIndex=$((${#MO_PARSED} - 20)) + moDots=... + + if [[ "$moIndex" -lt 0 ]]; then + moIndex=0 + moDots= + fi + + mo::escape moTemp "${MO_PARSED:$moIndex}" + moState="$moState parsed: $moDots$moTemp" + + moDots=... + + if [[ "${#MO_UNPARSED}" -le 20 ]]; then + moDots= + fi + + mo::escape moTemp "${MO_UNPARSED:0:20}$moDots" + moState="$moState unparsed: $moTemp" + + echo "DEBUG ${FUNCNAME[1]:-?} - $moState" >&2 +} + +# Internal: Show an error message and exit +# +# $1 - The error message to show +# $2 - Error code +# +# Returns nothing. Exits the program. +mo::error() { + echo "ERROR: $1" >&2 + exit "${2:-1}" +} + + +# Internal: Show an error message with a snippet of context and exit +# +# $1 - The error message to show +# $2 - The starting point +# $3 - Error code +# +# Returns nothing. Exits the program. +mo::errorNear() { + local moEscaped + + mo::escape moEscaped "${2:0:40}" + echo "ERROR: $1" >&2 + echo "ERROR STARTS NEAR: $moEscaped" + exit "${3:-1}" +} + + +# Internal: Displays the usage for mo. Pulls this from the file that +# contained the `mo` function. Can only work when the right filename +# comes is the one argument, and that only happens when `mo` is called +# with `$0` set to this file. +# +# $1 - Filename that has the help message +# +# Returns nothing. +mo::usage() { + while read -r line; do + if [[ "${line:0:2}" == "#/" ]]; then + echo "${line:3}" + fi + done < "$MO_ORIGINAL_COMMAND" + echo "" + echo "MO_VERSION=$MO_VERSION" +} + + +# Internal: Fetches the content to parse into MO_UNPARSED. Can be a list of +# partials for files or the content from stdin. +# +# $1 - Destination variable name +# $2-@ - File names (optional), read from stdin otherwise +# +# Returns nothing. +mo::content() { + local moTarget moContent moFilename + + moTarget=$1 + shift + moContent="" + + if [[ "${#@}" -gt 0 ]]; then + for moFilename in "$@"; do + mo::debug "Using template to load content from file: $moFilename" + #: This is so relative paths work from inside template files + moContent="$moContent$MO_OPEN_DELIMITER>$moFilename$MO_CLOSE_DELIMITER" + done + else + mo::debug "Will read content from stdin" + mo::contentFile moContent || return 1 + fi + + local "$moTarget" && mo::indirect "$moTarget" "$moContent" +} + + +# Internal: Read a file into MO_UNPARSED. +# +# $1 - Destination variable name. +# $2 - Filename to load - if empty, defaults to /dev/stdin +# +# Returns nothing. +mo::contentFile() { + local moFile moResult moContent + + #: The subshell removes any trailing newlines. We forcibly add + #: a dot to the content to preserve all newlines. Reading from + #: stdin with a `read` loop does not work as expected, so `cat` + #: needs to stay. + moFile=${2:-/dev/stdin} + + if [[ -e "$moFile" ]]; then + mo::debug "Loading content: $moFile" + moContent=$( + set +Ee + cat -- "$moFile" + moResult=$? + echo -n '.' + exit "$moResult" + ) || return 1 + moContent=${moContent%.} #: Remove last dot + elif [[ -n "${MO_FAIL_ON_FILE-}" ]]; then + mo::error "No such file: $moFile" + else + mo::debug "File does not exist: $moFile" + moContent="" + fi + + local "$1" && mo::indirect "$1" "$moContent" +} + + +# Internal: Send a variable up to the parent of the caller of this function. +# +# $1 - Variable name +# $2 - Value +# +# Examples +# +# callFunc () { +# local "$1" && mo::indirect "$1" "the value" +# } +# callFunc dest +# echo "$dest" # writes "the value" +# +# Returns nothing. +mo::indirect() { + unset -v "$1" + printf -v "$1" '%s' "$2" +} + + +# Internal: Send an array as a variable up to caller of a function +# +# $1 - Variable name +# $2-@ - Array elements +# +# Examples +# +# callFunc () { +# local myArray=(one two three) +# local "$1" && mo::indirectArray "$1" "${myArray[@]}" +# } +# callFunc dest +# echo "${dest[@]}" # writes "one two three" +# +# Returns nothing. +mo::indirectArray() { + unset -v "$1" + + #: IFS must be set to a string containing space or unset in order for + #: the array slicing to work regardless of the current IFS setting on + #: bash 3. This is detailed further at + #: https://github.com/fidian/gg-core/pull/7 + eval "$(printf "IFS= %s=(\"\${@:2}\") IFS=%q" "$1" "$IFS")" +} + + +# Internal: Trim leading characters from MO_UNPARSED +# +# Returns nothing. +mo::trimUnparsed() { + local moI moC + + moI=0 + moC=${MO_UNPARSED:0:1} + + while [[ "$moC" == " " || "$moC" == $'\r' || "$moC" == $'\n' || "$moC" == $'\t' ]]; do + moI=$((moI + 1)) + moC=${MO_UNPARSED:$moI:1} + done + + if [[ "$moI" != 0 ]]; then + MO_UNPARSED=${MO_UNPARSED:$moI} + fi +} + + +# Internal: Remove whitespace and content after whitespace +# +# $1 - Name of the destination variable +# $2 - The string to chomp +# +# Returns nothing. +mo::chomp() { + local moTemp moR moN moT + + moR=$'\r' + moN=$'\n' + moT=$'\t' + moTemp=${2%% *} + moTemp=${moTemp%%"$moR"*} + moTemp=${moTemp%%"$moN"*} + moTemp=${moTemp%%"$moT"*} + + local "$1" && mo::indirect "$1" "$moTemp" +} + + +# Public: Parses text, interpolates mustache tags. Utilizes the current value +# of MO_OPEN_DELIMITER, MO_CLOSE_DELIMITER, and MO_STANDALONE_CONTENT. Those +# three variables shouldn't be changed by user-defined functions. +# +# $1 - Destination variable name - where to store the finished content +# $2 - Content to parse +# $3 - Preserve standalone status/content - truthy if not empty. When set to a +# value, that becomes the standalone content value +# +# Returns nothing. +mo::parse() { + local moOldParsed moOldStandaloneContent moOldUnparsed moResult + + #: The standalone content is a trick to make the standalone tag detection + #: possible. When it's set to content with a newline and if the tag supports + #: it, the standalone content check happens. This check ensures only + #: whitespace is after the last newline up to the tag, and only whitespace + #: is after the tag up to the next newline. If that is the case, remove + #: whitespace and the trailing newline. By setting this to $'\n', we're + #: saying we are at the beginning of content. + mo::debug "Starting parse of ${#2} bytes" + moOldParsed=${MO_PARSED:-} + moOldUnparsed=${MO_UNPARSED:-} + MO_PARSED="" + MO_UNPARSED="$2" + + if [[ -z "${3:-}" ]]; then + moOldStandaloneContent=${MO_STANDALONE_CONTENT:-} + MO_STANDALONE_CONTENT=$'\n' + else + MO_STANDALONE_CONTENT=$3 + fi + + MO_CURRENT=${MO_CURRENT:-} + mo::parseInternal + moResult="$MO_PARSED$MO_UNPARSED" + MO_PARSED=$moOldParsed + MO_UNPARSED=$moOldUnparsed + + if [[ -z "${3:-}" ]]; then + MO_STANDALONE_CONTENT=$moOldStandaloneContent + fi + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Internal: Parse MO_UNPARSED, writing content to MO_PARSED. Interpolates +# mustache tags. +# +# No arguments +# +# Returns nothing. +mo::parseInternal() { + local moChunk + + mo::debug "Starting parse" + + while [[ -n "$MO_UNPARSED" ]]; do + mo::debugShowState + moChunk=${MO_UNPARSED%%"$MO_OPEN_DELIMITER"*} + MO_PARSED="$MO_PARSED$moChunk" + MO_STANDALONE_CONTENT="$MO_STANDALONE_CONTENT$moChunk" + MO_UNPARSED=${MO_UNPARSED:${#moChunk}} + + if [[ -n "$MO_UNPARSED" ]]; then + MO_UNPARSED=${MO_UNPARSED:${#MO_OPEN_DELIMITER}} + mo::trimUnparsed + + case "$MO_UNPARSED" in + '#'*) + #: Loop, if/then, or pass content through function + mo::parseBlock false + ;; + + '^'*) + #: Display section if named thing does not exist + mo::parseBlock true + ;; + + '>'*) + #: Load partial - get name of file relative to cwd + mo::parsePartial + ;; + + '/'*) + #: Closing tag + mo::errorNear "Unbalanced close tag" "$MO_UNPARSED" + ;; + + '!'*) + #: Comment - ignore the tag content entirely + mo::parseComment + ;; + + '='*) + #: Change delimiters + #: Any two non-whitespace sequences separated by whitespace. + mo::parseDelimiter + ;; + + '&'*) + #: Unescaped - mo doesn't escape/unescape + MO_UNPARSED=${MO_UNPARSED#&} + mo::trimUnparsed + mo::parseValue + ;; + + *) + #: Normal environment variable, string, subexpression, + #: current value, key, or function call + mo::parseValue + ;; + esac + fi + done +} + + +# Internal: Handle parsing a block +# +# $1 - Invert condition ("true" or "false") +# +# Returns nothing +mo::parseBlock() { + local moInvertBlock moTokens moTokensString + + moInvertBlock=$1 + MO_UNPARSED=${MO_UNPARSED:1} + mo::tokenizeTagContents moTokens "$MO_CLOSE_DELIMITER" + MO_UNPARSED=${MO_UNPARSED#"$MO_CLOSE_DELIMITER"} + mo::tokensToString moTokensString "${moTokens[@]:1}" + mo::debug "Parsing block: $moTokensString" + + if mo::standaloneCheck; then + mo::standaloneProcess + fi + + if [[ "${moTokens[1]}" == "NAME" ]] && mo::isFunction "${moTokens[2]}"; then + mo::parseBlockFunction "$moInvertBlock" "$moTokensString" "${moTokens[@]:1}" + elif [[ "${moTokens[1]}" == "NAME" ]] && mo::isArray "${moTokens[2]}"; then + mo::parseBlockArray "$moInvertBlock" "$moTokensString" "${moTokens[@]:1}" + else + mo::parseBlockValue "$moInvertBlock" "$moTokensString" "${moTokens[@]:1}" + fi +} + + +# Internal: Handle parsing a block whose first argument is a function +# +# $1 - Invert condition ("true" or "false") +# $2-@ - The parsed tokens from inside the block tags +# +# Returns nothing +mo::parseBlockFunction() { + local moTarget moInvertBlock moTokens moTemp moUnparsed moTokensString + + moInvertBlock=$1 + moTokensString=$2 + shift 2 + moTokens=(${@+"$@"}) + mo::debug "Parsing block function: $moTokensString" + mo::getContentUntilClose moTemp "$moTokensString" + #: Pass unparsed content to the function. + #: Keep the updated delimiters if they changed. + + if [[ "$moInvertBlock" != "true" ]]; then + mo::evaluateFunction moResult "$moTemp" "${moTokens[@]:1}" + MO_PARSED="$MO_PARSED$moResult" + fi + + mo::debug "Done parsing block function: $moTokensString" +} + + +# Internal: Handle parsing a block whose first argument is an array +# +# $1 - Invert condition ("true" or "false") +# $2-@ - The parsed tokens from inside the block tags +# +# Returns nothing +mo::parseBlockArray() { + local moInvertBlock moTokens moResult moArrayName moArrayIndexes moArrayIndex moTemp moUnparsed moOpenDelimiterBefore moCloseDelimiterBefore moOpenDelimiterAfter moCloseDelimiterAfter moParsed moTokensString moCurrent + + moInvertBlock=$1 + moTokensString=$2 + shift 2 + moTokens=(${@+"$@"}) + mo::debug "Parsing block array: $moTokensString" + moOpenDelimiterBefore=$MO_OPEN_DELIMITER + moCloseDelimiterBefore=$MO_CLOSE_DELIMITER + mo::getContentUntilClose moTemp "$moTokensString" + moOpenDelimiterAfter=$MO_OPEN_DELIMITER + moCloseDelimiterAfter=$MO_CLOSE_DELIMITER + moArrayName=${moTokens[1]} + eval "moArrayIndexes=(\"\${!${moArrayName}[@]}\")" + + if [[ "${#moArrayIndexes[@]}" -lt 1 ]]; then + #: No elements + if [[ "$moInvertBlock" == "true" ]]; then + #: Restore the delimiter before parsing + MO_OPEN_DELIMITER=$moOpenDelimiterBefore + MO_CLOSE_DELIMITER=$moCloseDelimiterBefore + moCurrent=$MO_CURRENT + MO_CURRENT=$moArrayName + mo::parse moParsed "$moTemp" "blockArrayInvert$MO_STANDALONE_CONTENT" + MO_CURRENT=$moCurrent + MO_PARSED="$MO_PARSED$moParsed" + fi + else + if [[ "$moInvertBlock" != "true" ]]; then + #: Process for each element in the array + moUnparsed=$MO_UNPARSED + + for moArrayIndex in "${moArrayIndexes[@]}"; do + #: Restore the delimiter before parsing + MO_OPEN_DELIMITER=$moOpenDelimiterBefore + MO_CLOSE_DELIMITER=$moCloseDelimiterBefore + moCurrent=$MO_CURRENT + MO_CURRENT=$moArrayName.$moArrayIndex + mo::debug "Iterate over array using element: $MO_CURRENT" + mo::parse moParsed "$moTemp" "blockArray$MO_STANDALONE_CONTENT" + MO_CURRENT=$moCurrent + MO_PARSED="$MO_PARSED$moParsed" + done + + MO_UNPARSED=$moUnparsed + fi + fi + + MO_OPEN_DELIMITER=$moOpenDelimiterAfter + MO_CLOSE_DELIMITER=$moCloseDelimiterAfter + mo::debug "Done parsing block array: $moTokensString" +} + + +# Internal: Handle parsing a block whose first argument is a value +# +# $1 - Invert condition ("true" or "false") +# $2-@ - The parsed tokens from inside the block tags +# +# Returns nothing +mo::parseBlockValue() { + local moInvertBlock moTokens moResult moUnparsed moOpenDelimiterBefore moOpenDelimiterAfter moCloseDelimiterBefore moCloseDelimiterAfter moParsed moTemp moTokensString moCurrent + + moInvertBlock=$1 + moTokensString=$2 + shift 2 + moTokens=(${@+"$@"}) + mo::debug "Parsing block value: $moTokensString" + moOpenDelimiterBefore=$MO_OPEN_DELIMITER + moCloseDelimiterBefore=$MO_CLOSE_DELIMITER + mo::getContentUntilClose moTemp "$moTokensString" + moOpenDelimiterAfter=$MO_OPEN_DELIMITER + moCloseDelimiterAfter=$MO_CLOSE_DELIMITER + + #: Variable, value, or list of mixed things + mo::evaluateListOfSingles moResult "${moTokens[@]}" + + if mo::isTruthy "$moResult" "$moInvertBlock"; then + mo::debug "Block is truthy: $moResult" + #: Restore the delimiter before parsing + MO_OPEN_DELIMITER=$moOpenDelimiterBefore + MO_CLOSE_DELIMITER=$moCloseDelimiterBefore + moCurrent=$MO_CURRENT + MO_CURRENT=${moTokens[1]} + mo::parse moParsed "$moTemp" "blockValue$MO_STANDALONE_CONTENT" + MO_PARSED="$MO_PARSED$moParsed" + MO_CURRENT=$moCurrent + fi + + MO_OPEN_DELIMITER=$moOpenDelimiterAfter + MO_CLOSE_DELIMITER=$moCloseDelimiterAfter + mo::debug "Done parsing block value: $moTokensString" +} + + +# Internal: Handle parsing a partial +# +# No arguments. +# +# Indentation will be applied to the entire partial's contents before parsing. +# This indentation is based on the whitespace that ends the previously parsed +# content. +# +# Returns nothing +mo::parsePartial() { + local moFilename moResult moIndentation moN moR moTemp moT + + MO_UNPARSED=${MO_UNPARSED:1} + mo::trimUnparsed + mo::chomp moFilename "${MO_UNPARSED%%"$MO_CLOSE_DELIMITER"*}" + MO_UNPARSED="${MO_UNPARSED#*"$MO_CLOSE_DELIMITER"}" + moIndentation="" + + if mo::standaloneCheck; then + moN=$'\n' + moR=$'\r' + moT=$'\t' + moIndentation="$moN${MO_PARSED//"$moR"/"$moN"}" + moIndentation=${moIndentation##*"$moN"} + moTemp=${moIndentation// } + moTemp=${moTemp//"$moT"} + + if [[ -n "$moTemp" ]]; then + moIndentation= + fi + + mo::debug "Adding indentation to partial: '$moIndentation'" + mo::standaloneProcess + fi + + mo::debug "Parsing partial: $moFilename" + + #: Execute in subshell to preserve current cwd and environment + moResult=$( + #: It would be nice to remove `dirname` and use a function instead, + #: but that is difficult when only given filenames. + cd "$(dirname -- "$moFilename")" || exit 1 + echo "$( + local moPartialContent moPartialParsed + + if ! mo::contentFile moPartialContent "${moFilename##*/}"; then + exit 1 + fi + + #: Reset delimiters before parsing + mo::indentLines moPartialContent "$moIndentation" "$moPartialContent" + MO_OPEN_DELIMITER="$MO_OPEN_DELIMITER_DEFAULT" + MO_CLOSE_DELIMITER="$MO_CLOSE_DELIMITER_DEFAULT" + mo::parse moPartialParsed "$moPartialContent" + + #: Fix bash handling of subshells and keep trailing whitespace. + echo -n "$moPartialParsed." + )" || exit 1 + ) || exit 1 + + if [[ -z "$moResult" ]]; then + mo::debug "Error detected when trying to read the file" + exit 1 + fi + + MO_PARSED="$MO_PARSED${moResult%.}" +} + + +# Internal: Handle parsing a comment +# +# No arguments. +# +# Returns nothing +mo::parseComment() { + local moContent moContent + + MO_UNPARSED=${MO_UNPARSED#*"$MO_CLOSE_DELIMITER"} + mo::debug "Parsing comment" + + if mo::standaloneCheck; then + mo::standaloneProcess + fi +} + + +# Internal: Handle parsing the change of delimiters +# +# No arguments. +# +# Returns nothing +mo::parseDelimiter() { + local moContent moOpen moClose + + MO_UNPARSED=${MO_UNPARSED:1} + mo::trimUnparsed + mo::chomp moOpen "$MO_UNPARSED" + MO_UNPARSED=${MO_UNPARSED:${#moOpen}} + mo::trimUnparsed + mo::chomp moClose "${MO_UNPARSED%%="$MO_CLOSE_DELIMITER"*}" + MO_UNPARSED=${MO_UNPARSED#*="$MO_CLOSE_DELIMITER"} + mo::debug "Parsing delimiters: $moOpen $moClose" + + if mo::standaloneCheck; then + mo::standaloneProcess + fi + + MO_OPEN_DELIMITER="$moOpen" + MO_CLOSE_DELIMITER="$moClose" +} + + +# Internal: Handle parsing value or function call +# +# No arguments. +# +# Returns nothing +mo::parseValue() { + local moUnparsedOriginal moTokens + + moUnparsedOriginal=$MO_UNPARSED + mo::tokenizeTagContents moTokens "$MO_CLOSE_DELIMITER" + mo::evaluate moResult "${moTokens[@]:1}" + MO_PARSED="$MO_PARSED$moResult" + + if [[ "${MO_UNPARSED:0:${#MO_CLOSE_DELIMITER}}" != "$MO_CLOSE_DELIMITER" ]]; then + mo::errorNear "Did not find closing tag" "$moUnparsedOriginal" + fi + + if mo::standaloneCheck; then + mo::standaloneProcess + fi + + MO_UNPARSED=${MO_UNPARSED:${#MO_CLOSE_DELIMITER}} +} + + +# Internal: Determine if the given name is a defined function. +# +# $1 - Function name to check +# +# Be extremely careful. Even if strict mode is enabled, it is not honored +# in newer versions of Bash. Any errors that crop up here will not be +# caught automatically. +# +# Examples +# +# moo () { +# echo "This is a function" +# } +# if mo::isFunction moo; then +# echo "moo is a defined function" +# fi +# +# Returns 0 if the name is a function, 1 otherwise. +mo::isFunction() { + local moFunctionName + + # Need to test for the array length, otherwise Mac will report an + # unbound variable + if [[ "${#MO_FUNCTION_CACHE_HIT[@]}" -gt 0 ]]; then + for moFunctionName in "${MO_FUNCTION_CACHE_HIT[@]}"; do + if [[ "$moFunctionName" == "$1" ]]; then + return 0 + fi + done + fi + + if [[ "${#MO_FUNCTION_CACHE_MISS[@]}" -gt 0 ]]; then + for moFunctionName in "${MO_FUNCTION_CACHE_MISS[@]}"; do + if [[ "$moFunctionName" == "$1" ]]; then + return 1 + fi + done + fi + + if declare -F "$1" &> /dev/null; then + if [[ "${#MO_FUNCTION_CACHE_HIT[@]}" -gt 0 ]]; then + MO_FUNCTION_CACHE_HIT=( ${MO_FUNCTION_CACHE_HIT[@]+"${MO_FUNCTION_CACHE_HIT[@]}"} "$1" ) + else + MO_FUNCTION_CACHE_HIT=( "$1" ) + fi + + return 0 + fi + + if [[ "${#MO_FUNCTION_CACHE_MISS[@]}" -gt 0 ]]; then + MO_FUNCTION_CACHE_MISS=( ${MO_FUNCTION_CACHE_MISS[@]+"${MO_FUNCTION_CACHE_MISS[@]}"} "$1" ) + else + MO_FUNCTION_CACHE_MISS=( "$1" ) + fi + + return 1 +} + + +# Internal: Determine if a given environment variable exists and if it is +# an array. +# +# $1 - Name of environment variable +# +# Be extremely careful. Even if strict mode is enabled, it is not honored +# in newer versions of Bash. Any errors that crop up here will not be +# caught automatically. +# +# Examples +# +# var=(abc) +# if moIsArray var; then +# echo "This is an array" +# echo "Make sure you don't accidentally use \$var" +# fi +# +# Returns 0 if the name is not empty, 1 otherwise. +mo::isArray() { + #: Namespace this variable so we don't conflict with what we're testing. + local moTestResult + + moTestResult=$(declare -p "$1" 2>/dev/null) || return 1 + [[ "${moTestResult:0:10}" == "declare -a" ]] && return 0 + [[ "${moTestResult:0:10}" == "declare -A" ]] && return 0 + + return 1 +} + + +# Internal: Determine if an array index exists. +# +# $1 - Variable name to check +# $2 - The index to check +# +# Has to check if the variable is an array and if the index is valid for that +# type of array. +# +# Returns true (0) if everything was ok, 1 if there's any condition that fails. +mo::isArrayIndexValid() { + local moDeclare moTest + + moDeclare=$(declare -p "$1") + moTest="" + + if [[ "${moDeclare:0:10}" == "declare -a" ]]; then + #: Numerically indexed array - must check if the index looks like a + #: number because using a string to index a numerically indexed array + #: will appear like it worked. + if [[ "$2" == "0" ]] || [[ "$2" =~ ^[1-9][0-9]*$ ]]; then + #: Index looks like a number + eval "moTest=\"\${$1[$2]+ok}\"" + fi + elif [[ "${moDeclare:0:10}" == "declare -A" ]]; then + #: Associative array + eval "moTest=\"\${$1[$2]+ok}\"" + fi + + if [[ -n "$moTest" ]]; then + return 0; + fi + + return 1 +} + + +# Internal: Determine if a variable is assigned, even if it is assigned an empty +# value. +# +# $1 - Variable name to check. +# +# Can not use logic like this in case invalid variable names are passed. +# [[ "${!1-a}" == "${!1-b}" ]] +# +# Using logic like this gives false positives. Also, this is not supported on +# Bash 3.2 and the script parsing will error before any commands are executed. +# [[ -v "$a" ]] +# +# Declaring a variable is not the same as assigning the variable. +# export x +# declare -p x # Output: declare -x x +# # Bash 3.2 returns error code 1 and outputs: +# # bash: declare: x: not found +# export y="" +# declare -p y # Output: declare -x y="" +# unset z +# declare -p z # Error code 1 and output: bash: declare: z: not found +# +# Returns true (0) if the variable is set, 1 if the variable is unset. +MO_VAR_TEST="ok" + +# This must not use `[[` because otherwise Bash 3.2 will stop execution and +# always write to stderr without the ability to capture it. +if test -v "MO_VAR_TEST" &> /dev/null; then + mo::debug "Using declare -p and [[ -v ]] for variable checks" + # More recent Bash + mo::isVarSet() { + # Do not convert this to [[, otherwise Bash 3.2 will fail to parse the + # script. + if declare -p "$1" &> /dev/null && test -v "$1"; then + return 0 + fi + + return 1 + } +else + mo::debug "Using declare -p for variable checks" + # Bash 3.2 + mo::isVarSet() { + # If the variable is exported and not assigned, declare -p will error. + if declare -p "$1" &> /dev/null; then + return 0 + fi + + return 1 + } +fi +unset MO_VAR_TEST + + +# Internal: Determine if a value is considered truthy. +# +# $1 - The value to test +# $2 - Invert the value, either "true" or "false" +# +# Returns true (0) if truthy, 1 otherwise. +mo::isTruthy() { + local moTruthy + + moTruthy=true + + if [[ -z "${1-}" ]]; then + moTruthy=false + elif [[ -n "${MO_FALSE_IS_EMPTY-}" ]] && [[ "${1-}" == "false" ]]; then + moTruthy=false + fi + + #: XOR the results + #: moTruthy inverse desiredResult + #: true false true + #: true true false + #: false false false + #: false true true + if [[ "$moTruthy" == "$2" ]]; then + mo::debug "Value is falsy, test result: $moTruthy inverse: $2" + return 1 + fi + + mo::debug "Value is truthy, test result: $moTruthy inverse: $2" + return 0 +} + + +# Internal: Convert token list to values +# +# $1 - Destination variable name +# $2-@ - Tokens to convert +# +# Sample call: +# +# mo::evaluate dest NAME username VALUE abc123 PAREN 2 +# +# Returns nothing. +mo::evaluate() { + local moTarget moStack moValue moType moIndex moCombined moResult + + moTarget=$1 + shift + + #: Phase 1 - remove all command tokens (PAREN, BRACE) + moStack=() + + while [[ $# -gt 0 ]]; do + case "$1" in + PAREN|BRACE) + moType=$1 + moValue=$2 + mo::debug "Combining $moValue tokens" + moIndex=$((${#moStack[@]} - (2 * moValue))) + mo::evaluateListOfSingles moCombined "${moStack[@]:$moIndex}" + + if [[ "$moType" == "PAREN" ]]; then + moStack=("${moStack[@]:0:$moIndex}" NAME "$moCombined") + else + moStack=("${moStack[@]:0:$moIndex}" VALUE "$moCombined") + fi + ;; + + *) + moStack=(${moStack[@]+"${moStack[@]}"} "$1" "$2") + ;; + esac + + shift 2 + done + + #: Phase 2 - check if this is a function or if we should just concatenate values + if [[ "${moStack[0]:-}" == "NAME" ]] && mo::isFunction "${moStack[1]}"; then + #: Special case - if the first argument is a function, then the rest are + #: passed to the function. + mo::debug "Evaluating function: ${moStack[1]}" + mo::evaluateFunction moResult "" "${moStack[@]:1}" + else + #: Concatenate + mo::debug "Concatenating ${#moStack[@]} stack items" + mo::evaluateListOfSingles moResult ${moStack[@]+"${moStack[@]}"} + fi + + local "$moTarget" && mo::indirect "$moTarget" "$moResult" +} + + +# Internal: Convert an argument list to individual values. +# +# $1 - Destination variable name +# $2-@ - A list of argument types and argument name/value. +# +# This assumes each value is separate from the rest. In contrast, mo::evaluate +# will pass all arguments to a function if the first value is a function. +# +# Sample call: +# +# mo::evaluateListOfSingles dest NAME username VALUE abc123 +# +# Returns nothing. +mo::evaluateListOfSingles() { + local moResult moTarget moTemp + + moTarget=$1 + shift + moResult="" + + while [[ $# -gt 1 ]]; do + mo::evaluateSingle moTemp "$1" "$2" + moResult="$moResult$moTemp" + shift 2 + done + + mo::debug "Evaluated list of singles: $moResult" + + local "$moTarget" && mo::indirect "$moTarget" "$moResult" +} + + +# Internal: Evaluate a single argument +# +# $1 - Name of variable for result +# $2 - Type of argument, either NAME or VALUE +# $3 - Argument +# +# Returns nothing +mo::evaluateSingle() { + local moResult moType moArg + + moType=$2 + moArg=$3 + mo::debug "Evaluating $moType: $moArg ($MO_CURRENT)" + + if [[ "$moType" == "VALUE" ]]; then + moResult=$moArg + elif [[ "$moArg" == "." ]]; then + mo::evaluateVariable moResult "" + elif [[ "$moArg" == "@key" ]]; then + mo::evaluateKey moResult + elif mo::isFunction "$moArg"; then + mo::evaluateFunction moResult "" "$moArg" + else + mo::evaluateVariable moResult "$moArg" + fi + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Internal: Return the value for @key based on current's name +# +# $1 - Name of variable for result +# +# Returns nothing +mo::evaluateKey() { + local moResult + + if [[ "$MO_CURRENT" == *.* ]]; then + moResult="${MO_CURRENT#*.}" + else + moResult="${MO_CURRENT}" + fi + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Internal: Handle a variable name +# +# $1 - Destination variable name +# $2 - Variable name +# +# Returns nothing. +mo::evaluateVariable() { + local moResult moArg moNameParts + + moArg=$2 + moResult="" + mo::findVariableName moNameParts "$moArg" + mo::debug "Evaluate variable ($moArg, $MO_CURRENT): ${moNameParts[*]}" + + if [[ -z "${moNameParts[1]}" ]]; then + if mo::isArray "${moNameParts[0]}"; then + eval mo::join moResult "," "\${${moNameParts[0]}[@]}" + else + if mo::isVarSet "${moNameParts[0]}"; then + moResult=${moNameParts[0]} + moResult="${!moResult}" + elif [[ -n "${MO_FAIL_ON_UNSET-}" ]]; then + mo::error "Environment variable not set: ${moNameParts[0]}" + fi + fi + else + if mo::isArray "${moNameParts[0]}"; then + eval "set +u;moResult=\"\${${moNameParts[0]}[${moNameParts[1]%%.*}]}\"" + else + mo::error "Unable to index a scalar as an array: $moArg" + fi + fi + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Internal: Find the name of a variable to use +# +# $1 - Destination variable name, receives an array +# $2 - Variable name from the template +# +# The array contains the following values +# [0] - Variable name +# [1] - Array index, or empty string +# +# Example variables +# a="a" +# b="b" +# c=("c.0" "c.1") +# d=([b]="d.b" [d]="d.d") +# +# Given these inputs (function input, current value), produce these outputs +# a c => a +# a c.0 => a +# b d => d.b +# b d.d => d.b +# a d => d.a +# a d.d => d.a +# c.0 d => c.0 +# d.b d => d.b +# '' c => c +# '' c.0 => c.0 +# Returns nothing. +mo::findVariableName() { + local moVar moNameParts moResultBase moResultIndex moCurrent + + moVar=$2 + moResultBase=$moVar + moResultIndex="" + + if [[ -z "$moVar" ]]; then + moResultBase=${MO_CURRENT%%.*} + + if [[ "$MO_CURRENT" == *.* ]]; then + moResultIndex=${MO_CURRENT#*.} + fi + elif [[ "$moVar" == *.* ]]; then + mo::debug "Find variable name; name has dot: $moVar" + moResultBase=${moVar%%.*} + moResultIndex=${moVar#*.} + elif [[ -n "$MO_CURRENT" ]]; then + moCurrent=${MO_CURRENT%%.*} + mo::debug "Find variable name; look in array: $moCurrent" + + if mo::isArrayIndexValid "$moCurrent" "$moVar"; then + moResultBase=$moCurrent + moResultIndex=$moVar + fi + fi + + local "$1" && mo::indirectArray "$1" "$moResultBase" "$moResultIndex" +} + + +# Internal: Join / implode an array +# +# $1 - Variable name to receive the joined content +# $2 - Joiner +# $3-@ - Elements to join +# +# Returns nothing. +mo::join() { + local joiner part result target + + target=$1 + joiner=$2 + result=$3 + shift 3 + + for part in "$@"; do + result="$result$joiner$part" + done + + local "$target" && mo::indirect "$target" "$result" +} + + +# Internal: Call a function. +# +# $1 - Variable for output +# $2 - Content to pass +# $3 - Function to call +# $4-@ - Additional arguments as list of type, value/name +# +# Returns nothing. +mo::evaluateFunction() { + local moArgs moContent moFunctionResult moTarget moFunction moTemp moFunctionCall + + moTarget=$1 + moContent=$2 + moFunction=$3 + shift 3 + moArgs=() + + while [[ $# -gt 1 ]]; do + mo::evaluateSingle moTemp "$1" "$2" + moArgs=(${moArgs[@]+"${moArgs[@]}"} "$moTemp") + shift 2 + done + + mo::escape moFunctionCall "$moFunction" + + if [[ -n "${MO_ALLOW_FUNCTION_ARGUMENTS-}" ]]; then + mo::debug "Function arguments are allowed" + + if [[ ${#moArgs[@]} -gt 0 ]]; then + for moTemp in "${moArgs[@]}"; do + mo::escape moTemp "$moTemp" + moFunctionCall="$moFunctionCall $moTemp" + done + fi + fi + + mo::debug "Calling function: $moFunctionCall" + + #: Call the function in a subshell for safety. Employ the trick to preserve + #: whitespace at the end of the output. + moContent=$( + export MO_FUNCTION_ARGS=(${moArgs[@]+"${moArgs[@]}"}) + echo -n "$moContent" | eval "$moFunctionCall ; moFunctionResult=\$? ; echo -n '.' ; exit \"\$moFunctionResult\"" + ) || { + moFunctionResult=$? + if [[ -n "${MO_FAIL_ON_FUNCTION-}" && "$moFunctionResult" != 0 ]]; then + mo::error "Function failed with status code $moFunctionResult: $moFunctionCall" "$moFunctionResult" + fi + } + + local "$moTarget" && mo::indirect "$moTarget" "${moContent%.}" +} + + +# Internal: Check if a tag appears to have only whitespace before it and after +# it on a line. There must be a new line before and there must be a newline +# after or the end of a string +# +# No arguments. +# +# Returns 0 if this is a standalone tag, 1 otherwise. +mo::standaloneCheck() { + local moContent moN moR moT + + moN=$'\n' + moR=$'\r' + moT=$'\t' + + #: Check the content before + moContent=${MO_STANDALONE_CONTENT//"$moR"/"$moN"} + + #: By default, signal to the next check that this one failed + MO_STANDALONE_CONTENT="" + + if [[ "$moContent" != *"$moN"* ]]; then + mo::debug "Not a standalone tag - no newline before" + + return 1 + fi + + moContent=${moContent##*"$moN"} + moContent=${moContent//"$moT"/} + moContent=${moContent// /} + + if [[ -n "$moContent" ]]; then + mo::debug "Not a standalone tag - non-whitespace detected before tag" + + return 1 + fi + + #: Check the content after + moContent=${MO_UNPARSED//"$moR"/"$moN"} + moContent=${moContent%%"$moN"*} + moContent=${moContent//"$moT"/} + moContent=${moContent// /} + + if [[ -n "$moContent" ]]; then + mo::debug "Not a standalone tag - non-whitespace detected after tag" + + return 1 + fi + + #: Signal to the next check that this tag removed content + MO_STANDALONE_CONTENT=$'\n' + + return 0 +} + + +# Internal: Process content before and after a tag. Remove prior whitespace up +# to the previous newline. Remove following whitespace up to and including the +# next newline. +# +# No arguments. +# +# Returns nothing. +mo::standaloneProcess() { + local moI moTemp + + mo::debug "Standalone tag - processing content before and after tag" + moI=$((${#MO_PARSED} - 1)) + mo::debug "zero done ${#MO_PARSED}" + mo::escape moTemp "$MO_PARSED" + mo::debug "$moTemp" + + # Mac appears to allow getting characters from before the start of the string + while [[ "$moI" -ge 0 ]] && [[ "${MO_PARSED:$moI:1}" == " " || "${MO_PARSED:$moI:1}" == $'\t' ]]; do + moI=$((moI - 1)) + done + + if [[ $((moI + 1)) != "${#MO_PARSED}" ]]; then + MO_PARSED="${MO_PARSED:0:${moI}+1}" + fi + + moI=0 + + while [[ "${MO_UNPARSED:${moI}:1}" == " " || "${MO_UNPARSED:${moI}:1}" == $'\t' ]]; do + moI=$((moI + 1)) + done + + if [[ "${MO_UNPARSED:${moI}:1}" == $'\r' ]]; then + moI=$((moI + 1)) + fi + + if [[ "${MO_UNPARSED:${moI}:1}" == $'\n' ]]; then + moI=$((moI + 1)) + fi + + if [[ "$moI" != 0 ]]; then + MO_UNPARSED=${MO_UNPARSED:${moI}} + fi +} + + +# Internal: Apply indentation before any line that has content in MO_UNPARSED. +# +# $1 - Destination variable name. +# $2 - The indentation string. +# $3 - The content that needs the indentation string prepended on each line. +# +# Returns nothing. +mo::indentLines() { + local moContent moIndentation moResult moN moR moChunk + + moIndentation=$2 + moContent=$3 + + if [[ -z "$moIndentation" ]]; then + mo::debug "Not applying indentation, empty indentation" + + local "$1" && mo::indirect "$1" "$moContent" + return + fi + + if [[ -z "$moContent" ]]; then + mo::debug "Not applying indentation, empty contents" + + local "$1" && mo::indirect "$1" "$moContent" + return + fi + + moResult= + moN=$'\n' + moR=$'\r' + + mo::debug "Applying indentation: '${moIndentation}'" + + while [[ -n "$moContent" ]]; do + moChunk=${moContent%%"$moN"*} + moChunk=${moChunk%%"$moR"*} + moContent=${moContent:${#moChunk}} + + if [[ -n "$moChunk" ]]; then + moResult="$moResult$moIndentation$moChunk" + fi + + moResult="$moResult${moContent:0:1}" + moContent=${moContent:1} + done + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Internal: Escape a value +# +# $1 - Destination variable name +# $2 - Value to escape +# +# Returns nothing +mo::escape() { + local moResult + + moResult=$2 + moResult=$(declare -p moResult) + moResult=${moResult#*=} + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Internal: Get the content up to the end of the block by minimally parsing and +# balancing blocks. Returns the content before the end tag to the caller and +# removes the content + the end tag from MO_UNPARSED. This can change the +# delimiters, adjusting MO_OPEN_DELIMITER and MO_CLOSE_DELIMITER. +# +# $1 - Destination variable name +# $2 - Token string to match for a closing tag +# +# Returns nothing. +mo::getContentUntilClose() { + local moChunk moResult moTemp moTokensString moTokens moTarget moTagStack moResultTemp + + moTarget=$1 + moTagStack=("$2") + mo::debug "Get content until close tag: ${moTagStack[0]}" + moResult="" + + while [[ -n "$MO_UNPARSED" ]] && [[ "${#moTagStack[@]}" -gt 0 ]]; do + moChunk=${MO_UNPARSED%%"$MO_OPEN_DELIMITER"*} + moResult="$moResult$moChunk" + MO_UNPARSED=${MO_UNPARSED:${#moChunk}} + + if [[ -n "$MO_UNPARSED" ]]; then + moResultTemp="$MO_OPEN_DELIMITER" + MO_UNPARSED=${MO_UNPARSED:${#MO_OPEN_DELIMITER}} + mo::getContentTrim moTemp + moResultTemp="$moResultTemp$moTemp" + mo::debug "First character within tag: ${MO_UNPARSED:0:1}" + + case "$MO_UNPARSED" in + '#'*) + #: Increase block + moResultTemp="$moResultTemp${MO_UNPARSED:0:1}" + MO_UNPARSED=${MO_UNPARSED:1} + mo::getContentTrim moTemp + mo::getContentWithinTag moTemp "$MO_CLOSE_DELIMITER" + moResultTemp="$moResultTemp${moTemp[0]}" + moTagStack=("${moTemp[1]}" "${moTagStack[@]}") + ;; + + '^'*) + #: Increase block + moResultTemp="$moResultTemp${MO_UNPARSED:0:1}" + MO_UNPARSED=${MO_UNPARSED:1} + mo::getContentTrim moTemp + mo::getContentWithinTag moTemp "$MO_CLOSE_DELIMITER" + moResultTemp="$moResultTemp${moTemp[0]}" + moTagStack=("${moTemp[1]}" "${moTagStack[@]}") + ;; + + '>'*) + #: Partial - ignore + moResultTemp="$moResultTemp${MO_UNPARSED:0:1}" + MO_UNPARSED=${MO_UNPARSED:1} + mo::getContentTrim moTemp + mo::getContentWithinTag moTemp "$MO_CLOSE_DELIMITER" + moResultTemp="$moResultTemp${moTemp[0]}" + ;; + + '/'*) + #: Decrease block + moResultTemp="$moResultTemp${MO_UNPARSED:0:1}" + MO_UNPARSED=${MO_UNPARSED:1} + mo::getContentTrim moTemp + mo::getContentWithinTag moTemp "$MO_CLOSE_DELIMITER" + + if [[ "${moTagStack[0]}" == "${moTemp[1]}" ]]; then + moResultTemp="$moResultTemp${moTemp[0]}" + moTagStack=("${moTagStack[@]:1}") + + if [[ "${#moTagStack[@]}" -eq 0 ]]; then + #: Erase all portions of the close tag + moResultTemp="" + fi + else + mo::errorNear "Unbalanced closing tag, expected: ${moTagStack[0]}" "${moTemp[0]}${MO_UNPARSED}" + fi + ;; + + '!'*) + #: Comment - ignore + mo::getContentComment moTemp + moResultTemp="$moResultTemp$moTemp" + ;; + + '='*) + #: Change delimiters + mo::getContentDelimiter moTemp + moResultTemp="$moResultTemp$moTemp" + ;; + + '&'*) + #: Unescaped - bypass one then ignore + moResultTemp="$moResultTemp${MO_UNPARSED:0:1}" + MO_UNPARSED=${MO_UNPARSED:1} + mo::getContentTrim moTemp + moResultTemp="$moResultTemp$moTemp" + mo::getContentWithinTag moTemp "$MO_CLOSE_DELIMITER" + moResultTemp="$moResultTemp${moTemp[0]}" + ;; + + *) + #: Normal variable - ignore + mo::getContentWithinTag moTemp "$MO_CLOSE_DELIMITER" + moResultTemp="$moResultTemp${moTemp[0]}" + ;; + esac + + moResult="$moResult$moResultTemp" + fi + done + + MO_STANDALONE_CONTENT="$MO_STANDALONE_CONTENT$moResult" + + if mo::standaloneCheck; then + moResultTemp=$MO_PARSED + MO_PARSED=$moResult + mo::standaloneProcess + moResult=$MO_PARSED + MO_PARSED=$moResultTemp + fi + + local "$moTarget" && mo::indirect "$moTarget" "$moResult" +} + + +# Internal: Convert a list of tokens to a string +# +# $1 - Destination variable for the string +# $2-$@ - Token list +# +# Returns nothing. +mo::tokensToString() { + local moTarget moString moTokens + + moTarget=$1 + shift 1 + moTokens=("$@") + moString=$(declare -p moTokens) + moString=${moString#*=} + + local "$moTarget" && mo::indirect "$moTarget" "$moString" +} + + +# Internal: Trims content from MO_UNPARSED, returns trimmed content. +# +# $1 - Destination variable +# +# Returns nothing. +mo::getContentTrim() { + local moChar moResult + + moChar=${MO_UNPARSED:0:1} + moResult="" + + while [[ "$moChar" == " " ]] || [[ "$moChar" == $'\r' ]] || [[ "$moChar" == $'\t' ]] || [[ "$moChar" == $'\n' ]]; do + moResult="$moResult$moChar" + MO_UNPARSED=${MO_UNPARSED:1} + moChar=${MO_UNPARSED:0:1} + done + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Get the content up to and including a close tag +# +# $1 - Destination variable +# +# Returns nothing. +mo::getContentComment() { + local moResult + + mo::debug "Getting content for comment" + moResult=${MO_UNPARSED%%"$MO_CLOSE_DELIMITER"*} + MO_UNPARSED=${MO_UNPARSED:${#moResult}} + + if [[ "$MO_UNPARSED" == "$MO_CLOSE_DELIMITER"* ]]; then + moResult="$moResult$MO_CLOSE_DELIMITER" + MO_UNPARSED=${MO_UNPARSED#"$MO_CLOSE_DELIMITER"} + fi + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Get the content up to and including a close tag. First two non-whitespace +# tokens become the new open and close tag. +# +# $1 - Destination variable +# +# Returns nothing. +mo::getContentDelimiter() { + local moResult moTemp moOpen moClose + + mo::debug "Getting content for delimiter" + moResult="" + mo::getContentTrim moTemp + moResult="$moResult$moTemp" + mo::chomp moOpen "$MO_UNPARSED" + MO_UNPARSED="${MO_UNPARSED:${#moOpen}}" + moResult="$moResult$moOpen" + mo::getContentTrim moTemp + moResult="$moResult$moTemp" + mo::chomp moClose "${MO_UNPARSED%%="$MO_CLOSE_DELIMITER"*}" + MO_UNPARSED="${MO_UNPARSED:${#moClose}}" + moResult="$moResult$moClose" + mo::getContentTrim moTemp + moResult="$moResult$moTemp" + MO_OPEN_DELIMITER="$moOpen" + MO_CLOSE_DELIMITER="$moClose" + + local "$1" && mo::indirect "$1" "$moResult" +} + + +# Get the content up to and including a close tag. First two non-whitespace +# tokens become the new open and close tag. +# +# $1 - Destination variable, an array +# $2 - Terminator string +# +# The array contents: +# [0] The raw content within the tag +# [1] The parsed tokens as a single string +# +# Returns nothing. +mo::getContentWithinTag() { + local moUnparsed moTokens + + moUnparsed=${MO_UNPARSED} + mo::tokenizeTagContents moTokens "$MO_CLOSE_DELIMITER" + MO_UNPARSED=${MO_UNPARSED#"$MO_CLOSE_DELIMITER"} + mo::tokensToString moTokensString "${moTokens[@]:1}" + moParsed=${moUnparsed:0:$((${#moUnparsed} - ${#MO_UNPARSED}))} + + local "$1" && mo::indirectArray "$1" "$moParsed" "$moTokensString" +} + + +# Internal: Parse MO_UNPARSED and retrieve the content within the tag +# delimiters. Converts everything into an array of string values. +# +# $1 - Destination variable for the array of contents. +# $2 - Stop processing when this content is found. +# +# The list of tokens are in RPN form. The first item in the resulting array is +# the number of actual tokens (after combining command tokens) in the list. +# +# Given: a 'bc' "de\"\n" (f {g 'h'}) +# Result: ([0]=4 [1]=NAME [2]=a [3]=VALUE [4]=bc [5]=VALUE [6]=$'de\"\n' +# [7]=NAME [8]=f [9]=NAME [10]=g [11]=VALUE [12]=h +# [13]=BRACE [14]=2 [15]=PAREN [16]=2 +# +# Returns nothing +mo::tokenizeTagContents() { + local moResult moTerminator moTemp moUnparsedOriginal moTokenCount + + moTerminator=$2 + moResult=() + moUnparsedOriginal=$MO_UNPARSED + moTokenCount=0 + mo::debug "Tokenizing tag contents until terminator: $moTerminator" + + while true; do + mo::trimUnparsed + + case "$MO_UNPARSED" in + "") + mo::errorNear "Did not find matching terminator: $moTerminator" "$moUnparsedOriginal" + ;; + + "$moTerminator"*) + mo::debug "Found terminator" + local "$1" && mo::indirectArray "$1" "$moTokenCount" ${moResult[@]+"${moResult[@]}"} + return + ;; + + '('*) + #: Do not tokenize the open paren - treat this as RPL + MO_UNPARSED=${MO_UNPARSED:1} + mo::tokenizeTagContents moTemp ')' + moResult=(${moResult[@]+"${moResult[@]}"} "${moTemp[@]:1}" PAREN "${moTemp[0]}") + MO_UNPARSED=${MO_UNPARSED:1} + ;; + + '{'*) + #: Do not tokenize the open brace - treat this as RPL + MO_UNPARSED=${MO_UNPARSED:1} + mo::tokenizeTagContents moTemp '}' + moResult=(${moResult[@]+"${moResult[@]}"} "${moTemp[@]:1}" BRACE "${moTemp[0]}") + MO_UNPARSED=${MO_UNPARSED:1} + ;; + + ')'* | '}'*) + mo::errorNear "Unbalanced closing parenthesis or brace" "$MO_UNPARSED" + ;; + + "'"*) + mo::tokenizeTagContentsSingleQuote moTemp + moResult=(${moResult[@]+"${moResult[@]}"} "${moTemp[@]}") + ;; + + '"'*) + mo::tokenizeTagContentsDoubleQuote moTemp + moResult=(${moResult[@]+"${moResult[@]}"} "${moTemp[@]}") + ;; + + *) + mo::tokenizeTagContentsName moTemp + moResult=(${moResult[@]+"${moResult[@]}"} "${moTemp[@]}") + ;; + esac + + mo::debug "Got chunk: ${moTemp[0]} ${moTemp[1]}" + moTokenCount=$((moTokenCount + 1)) + done +} + + +# Internal: Get the contents of a variable name. +# +# $1 - Destination variable name for the token list (array of strings) +# +# Returns nothing +mo::tokenizeTagContentsName() { + local moTemp + + mo::chomp moTemp "${MO_UNPARSED%%"$MO_CLOSE_DELIMITER"*}" + moTemp=${moTemp%%(*} + moTemp=${moTemp%%)*} + moTemp=${moTemp%%\{*} + moTemp=${moTemp%%\}*} + MO_UNPARSED=${MO_UNPARSED:${#moTemp}} + mo::trimUnparsed + mo::debug "Parsed default token: $moTemp" + + local "$1" && mo::indirectArray "$1" "NAME" "$moTemp" +} + + +# Internal: Get the contents of a tag in double quotes. Parses the backslash +# sequences. +# +# $1 - Destination variable name for the token list (array of strings) +# +# Returns nothing. +mo::tokenizeTagContentsDoubleQuote() { + local moResult moUnparsedOriginal + + moUnparsedOriginal=$MO_UNPARSED + MO_UNPARSED=${MO_UNPARSED:1} + moResult= + mo::debug "Getting double quoted tag contents" + + while true; do + if [[ -z "$MO_UNPARSED" ]]; then + mo::errorNear "Unbalanced double quote" "$moUnparsedOriginal" + fi + + case "$MO_UNPARSED" in + '"'*) + MO_UNPARSED=${MO_UNPARSED:1} + local "$1" && mo::indirectArray "$1" "VALUE" "$moResult" + return + ;; + + \\b*) + moResult="$moResult"$'\b' + MO_UNPARSED=${MO_UNPARSED:2} + ;; + + \\e*) + #: Note, \e is ESC, but in Bash $'\E' is ESC. + moResult="$moResult"$'\E' + MO_UNPARSED=${MO_UNPARSED:2} + ;; + + \\f*) + moResult="$moResult"$'\f' + MO_UNPARSED=${MO_UNPARSED:2} + ;; + + \\n*) + moResult="$moResult"$'\n' + MO_UNPARSED=${MO_UNPARSED:2} + ;; + + \\r*) + moResult="$moResult"$'\r' + MO_UNPARSED=${MO_UNPARSED:2} + ;; + + \\t*) + moResult="$moResult"$'\t' + MO_UNPARSED=${MO_UNPARSED:2} + ;; + + \\v*) + moResult="$moResult"$'\v' + MO_UNPARSED=${MO_UNPARSED:2} + ;; + + \\*) + moResult="$moResult${MO_UNPARSED:1:1}" + MO_UNPARSED=${MO_UNPARSED:2} + ;; + + *) + moResult="$moResult${MO_UNPARSED:0:1}" + MO_UNPARSED=${MO_UNPARSED:1} + ;; + esac + done +} + + +# Internal: Get the contents of a tag in single quotes. Only gets the raw +# value. +# +# $1 - Destination variable name for the token list (array of strings) +# +# Returns nothing. +mo::tokenizeTagContentsSingleQuote() { + local moResult moUnparsedOriginal + + moUnparsedOriginal=$MO_UNPARSED + MO_UNPARSED=${MO_UNPARSED:1} + moResult= + mo::debug "Getting single quoted tag contents" + + while true; do + if [[ -z "$MO_UNPARSED" ]]; then + mo::errorNear "Unbalanced single quote" "$moUnparsedOriginal" + fi + + case "$MO_UNPARSED" in + "'"*) + MO_UNPARSED=${MO_UNPARSED:1} + local "$1" && mo::indirectArray "$1" VALUE "$moResult" + return + ;; + + *) + moResult="$moResult${MO_UNPARSED:0:1}" + MO_UNPARSED=${MO_UNPARSED:1} + ;; + esac + done +} + + +# Save the original command's path for usage later +MO_ORIGINAL_COMMAND="$(cd "${BASH_SOURCE[0]%/*}" || exit 1; pwd)/${BASH_SOURCE[0]##*/}" +MO_VERSION="3.1.0" + +# If sourced, load all functions. +# If executed, perform the actions as expected. +if [[ "$0" == "${BASH_SOURCE[0]}" ]] || [[ -z "${BASH_SOURCE[0]}" ]]; then + mo "$@" +fi diff --git a/build/templates/home.html b/build/templates/home.html new file mode 100644 index 0000000..e0b7521 --- /dev/null +++ b/build/templates/home.html @@ -0,0 +1,12 @@ +<!DOCTYPE html> +<html lang="en"> + <head> + {{> head.html}} + <title>ubergeek</title> + </head> + <body> + {{> header.html}} + <main>{{{CONTENT}}}</main> + {{> footer.html}} + </body> +</html> diff --git a/build/templates/page.html b/build/templates/page.html new file mode 100644 index 0000000..502dd58 --- /dev/null +++ b/build/templates/page.html @@ -0,0 +1,12 @@ +<!DOCTYPE html> +<html lang="{{LANG}}"> + <head> + {{> head.html}} + <title>ubergeek{{#PAGE_TITLE}}: {{PAGE_TITLE}}{{/PAGE_TITLE}}</title> + </head> + <body{{#DIR}} dir="{{DIR}}"{{/DIR}}> + {{> header.html}} + <main>{{{CONTENT}}}</main> + {{> footer.html}} + </body> +</html> diff --git a/build/templates/partials/.gitkeep b/build/templates/partials/.gitkeep new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/build/templates/partials/.gitkeep diff --git a/build/templates/partials/footer.html b/build/templates/partials/footer.html new file mode 100644 index 0000000..6dca778 --- /dev/null +++ b/build/templates/partials/footer.html @@ -0,0 +1,16 @@ +<footer> + <hr /> + <a href="https://git.gumx.cc">git</a> / + <a href="https://mail.gumx.cc">mail</a> / + <a href="https://irc.gumx.cc">irc</a> / + <a href="https://vpn.gumx.cc">vpn</a> / + <a href="https://pgp.gumx.cc">pgp</a> / + <a href="https://wk.fo">wk.fo</a> + <br /> + <a href="https://webring.gumx.cc/gumx.cc/previous">← previous site</a> / + <a href="https://webring.gumx.cc/">webring</a> / + <a href="https://webring.gumx.cc/gumx.cc/next">next site →</a> + <br /> + <a href="https://git.gumx.cc/ubergeek">source</a> / + <a href="/license">license</a> +</footer> diff --git a/build/templates/partials/head.html b/build/templates/partials/head.html new file mode 100644 index 0000000..f9f143e --- /dev/null +++ b/build/templates/partials/head.html @@ -0,0 +1,11 @@ + <meta charset="utf-8" /> + <meta name="viewport" content="width=device-width,initial-scale=1" /> + <meta name="author" content="Ahmed (gumx) Alaa" /> + <meta name="copyright" content="CC BY-SA 4.0" /> + {{#META_DATE}}<meta name="date" content="{{META_DATE}}" />{{/META_DATE}} + {{#META_DESCRIPTION}}<meta name="description" content="{{META_DESCRIPTION}}" />{{/META_DESCRIPTION}} + <link rel="stylesheet" href="/styles/main.css" /> + {{#HAS_CODE}}<link rel="stylesheet" href="/styles/highlight.min.css" /> + <script src="/scripts/highlight.min.js"></script> + <script>hljs.highlightAll();</script>{{/HAS_CODE}} + <link href="data:," rel="icon"> diff --git a/build/templates/partials/header.html b/build/templates/partials/header.html new file mode 100644 index 0000000..cbe3936 --- /dev/null +++ b/build/templates/partials/header.html @@ -0,0 +1,6 @@ + <header> + <nav> + <strong><a href="/">ubergeek</a>{{BREADCRUMBS}}</strong> + {{#HEADER_EXTRA}}<br /><em>{{HEADER_EXTRA}}</em>{{/HEADER_EXTRA}} + </nav> + </header> diff --git a/linux-command-line/shell-basics/index.md b/linux-command-line/shell-basics/index.md new file mode 100644 index 0000000..c0cf75b --- /dev/null +++ b/linux-command-line/shell-basics/index.md @@ -0,0 +1,327 @@ +--- +title: Shell Basics +--- + +# Shell Basics + +The shell is your primary interface to a Linux system. Everything — file management, process control, system administration — flows through it. Before graphical tools, before IDEs, there was the shell. Mastering it is non-negotiable. + +**Prerequisites:** None +**Estimated time:** 4-6 hours of practice + +--- + +## What is a Shell? + +A shell is a program that reads commands from your input and executes them. It's both a **command interpreter** (runs programs) and a **programming language** (scripts). + +Common shells: +- **bash** — Bourne Again Shell, the default on most Linux systems +- **zsh** — Extended bash with better completion and prompting +- **sh** — POSIX shell, the lowest common denominator +- **fish** — Friendly interactive shell, not POSIX-compatible + +When you open a terminal, you're running a **terminal emulator** (alacritty, kitty, xterm) which hosts a **shell process**. The terminal draws pixels; the shell interprets commands. + +```bash +# Check your current shell +echo $SHELL + +# List available shells +cat /etc/shells + +# The shell prompt typically shows: +# username@hostname:current_directory$ +``` + +> **Key distinction:** The terminal and the shell are different things. You can run bash inside alacritty, or inside a Linux TTY, or over SSH. The shell doesn't care what's displaying its output. + +--- + +## Navigating the Filesystem + +Linux has a single filesystem tree rooted at `/`. Everything is a file — devices, processes, sockets. + +### Essential Navigation Commands + +```bash +pwd # Print working directory — where you are +ls # List files in current directory +ls -la # Long format, show hidden files (dotfiles) +ls -lh # Human-readable sizes (K, M, G) +cd /etc # Change to /etc +cd ~ # Change to home directory (also just: cd) +cd - # Change to previous directory +cd .. # Go up one level +``` + +### The Filesystem Hierarchy + +| Path | Purpose | +|------|---------| +| `/` | Root of everything | +| `/home` | User home directories | +| `/etc` | System configuration files | +| `/var` | Variable data (logs, databases, mail) | +| `/tmp` | Temporary files (cleared on reboot) | +| `/usr` | User programs and data (read-only) | +| `/usr/bin` | Most user commands | +| `/usr/local` | Locally installed software | +| `/dev` | Device files | +| `/proc` | Process information pseudo-filesystem | +| `/sys` | Kernel/device information pseudo-filesystem | +| `/opt` | Optional/third-party software | + +```bash +# The filesystem is a tree. Explore it: +ls / +ls /etc +ls /dev + +# Everything is a file: +file /dev/null # character special device +file /etc/hostname # ASCII text +file /usr/bin/ls # ELF 64-bit executable +``` + +--- + +## File Operations + +### Creating, Copying, Moving, Deleting + +```bash +# Create +touch newfile.txt # Create empty file (or update timestamp) +mkdir mydir # Create directory +mkdir -p path/to/nested/dir # Create nested directories + +# Copy +cp file.txt backup.txt # Copy file +cp -r mydir/ mydir_backup/ # Copy directory recursively + +# Move / Rename +mv old.txt new.txt # Rename +mv file.txt ~/Documents/ # Move to another directory + +# Delete +rm file.txt # Remove file (no trash, no undo) +rm -r mydir/ # Remove directory recursively +rm -ri mydir/ # Interactive — asks before each deletion +rmdir emptydir/ # Remove only if empty +``` + +> **Warning:** `rm` is permanent. There is no trash can. `rm -rf /` will destroy your entire system if you have the permissions. Always double-check your path before pressing Enter. + +### Viewing Files + +```bash +cat file.txt # Print entire file to stdout +less file.txt # Paginated viewer (q to quit, / to search) +head -n 20 file.txt # First 20 lines +tail -n 20 file.txt # Last 20 lines +tail -f /var/log/syslog # Follow a file in real-time (great for logs) +wc -l file.txt # Count lines +``` + +`less` is your friend. Use it for anything longer than a screenful. Key bindings inside `less`: +- `Space` / `b` — page down / up +- `/pattern` — search forward +- `n` / `N` — next / previous search match +- `g` / `G` — go to top / bottom +- `q` — quit + +--- + +## Permissions + +Every file has an **owner**, a **group**, and a set of **permissions** (read, write, execute) for each of: owner, group, others. + +```bash +ls -l file.txt +# -rw-r--r-- 1 ahmed users 1234 Mar 31 10:00 file.txt +# │├──┤├──┤├──┤ +# │ │ │ └── others: read only +# │ │ └────── group: read only +# │ └─────────── owner: read + write +# └───────────── file type: - = regular, d = directory, l = symlink +``` + +### chmod — Change Permissions + +```bash +# Symbolic notation +chmod u+x script.sh # Add execute for owner +chmod go-w file.txt # Remove write for group and others +chmod a+r file.txt # Add read for all + +# Octal notation (most common) +chmod 755 script.sh # rwxr-xr-x — owner full, others read+execute +chmod 644 file.txt # rw-r--r-- — owner read+write, others read +chmod 600 secret.key # rw------- — owner only +``` + +**Octal cheat sheet:** r=4, w=2, x=1. Add them up per group. +- `7` = rwx (4+2+1) +- `6` = rw- (4+2) +- `5` = r-x (4+1) +- `4` = r-- (4) +- `0` = --- (0) + +### chown — Change Ownership + +```bash +chown ahmed file.txt # Change owner +chown ahmed:users file.txt # Change owner and group +chown -R ahmed:users mydir/ # Recursive +``` + +### Directories and Execute Permission + +The execute bit on directories means something different: it grants **access** (the ability to `cd` into it and access files within). A directory with `r--` lets you list filenames but not read the files. A directory with `--x` lets you access files by name but not list them. + +--- + +## Pipes and Redirection + +This is where the shell gets powerful. The Unix philosophy: small programs that do one thing well, connected together. + +### Redirection + +Every process has three standard streams: +- **stdin** (0) — input +- **stdout** (1) — normal output +- **stderr** (2) — error output + +```bash +# Redirect stdout to a file +ls > filelist.txt # Overwrite +ls >> filelist.txt # Append + +# Redirect stderr +gcc broken.c 2> errors.txt # Errors go to file +gcc broken.c 2>&1 # Stderr merges into stdout + +# Redirect both +command > output.txt 2>&1 # Both to same file +command &> output.txt # Bash shorthand for the above + +# Discard output +command > /dev/null 2>&1 # Silence everything + +# Read stdin from a file +sort < unsorted.txt +``` + +### Pipes + +Pipes connect stdout of one command to stdin of the next. + +```bash +# Find the 10 largest files in /var/log +du -sh /var/log/* 2>/dev/null | sort -rh | head -10 + +# Count unique IP addresses in a log +cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20 + +# Find running processes matching a pattern +ps aux | grep nginx | grep -v grep + +# Chain as many as you need +cat /etc/passwd | cut -d: -f1 | sort | head -5 +``` + +> **Useless use of cat:** `cat file | grep pattern` can be written as `grep pattern file`. But in pipelines with many stages, starting with `cat` can be clearer. Don't stress about it early on — clarity matters more than micro-optimization. + +### Command Substitution + +Use the output of one command inside another: + +```bash +# Using $() — preferred +echo "Today is $(date +%Y-%m-%d)" +files=$(ls *.txt) + +# Using backticks — older style, harder to nest +echo "Today is `date +%Y-%m-%d`" +``` + +--- + +## Wildcards (Globbing) + +The shell expands patterns before passing them to commands: + +```bash +* # Match any characters (except leading dot) +? # Match exactly one character +[abc] # Match a, b, or c +[0-9] # Match any digit +[!abc] # Match anything except a, b, c +** # Match directories recursively (bash: shopt -s globstar) + +# Examples +ls *.txt # All .txt files +ls file?.log # file1.log, fileA.log, etc. +ls *.{jpg,png} # All .jpg and .png files (brace expansion, not globbing) +rm /tmp/test_* # All files starting with test_ in /tmp +``` + +--- + +## Getting Help + +```bash +man ls # Manual page for ls (press q to quit) +man -k "copy files" # Search manual descriptions +ls --help # Quick help (most GNU tools) +type ls # Is it a builtin, alias, or binary? +which python # Full path of a command +whatis ls # One-line description +apropos network # Search man pages by keyword +``` + +**Reading man pages:** The synopsis section uses conventions: +- `[optional]` — square brackets mean optional +- `REQUIRED` — caps or no brackets means required +- `...` — can be repeated +- `a | b` — choose one + +--- + +## Exercises + +### Drill Exercises + +1. Navigate to `/etc` and list all files starting with `host`. What do they contain? +2. Create a directory structure: `~/practice/project/{src,docs,tests}` in one command +3. Create a file, set its permissions to 750, then explain who can do what with it +4. Use `ls -la /` and identify which entries are regular files, directories, and symlinks +5. Redirect the output of `ls /nonexistent` to `/dev/null` while keeping error messages visible + +### Pipeline Challenges + +6. Count how many lines in `/etc/passwd` contain the string `nologin` +7. List the 5 largest files in `/usr/bin` by size (hint: `ls -lS` or `du`) +8. Extract just the usernames from `/etc/passwd` (field 1, colon-separated) and sort them +9. Find all environment variables that contain the string `PATH` (hint: `env | grep`) +10. Get the total line count of all `.conf` files in `/etc/` (hint: `wc -l`, `cat`) + +### Mini-Project + +**Build a quick reference card:** Create a file `~/cheatsheet.md` that contains the 20 commands you found most useful from this section, organized by category, with a one-line description and example for each. This is your cheatsheet — you'll extend it throughout the course. + +--- + +## Key Takeaways + +- The shell is a command interpreter and a programming environment +- Everything in Linux is a file, organized in a single tree from `/` +- Permissions are the foundation of Linux security — understand `rwx` and octal +- Pipes and redirection are the core of composing commands +- `man` and `--help` are always available — read them first, search the web second +- Practice in a real terminal. Reading about commands is not the same as using them. + +--- + +**Next:** [File Management](../file-management/) — find, locate, tar, rsync, and more |