Real_Author : Luo Hao
Note : Since the project was launched to the 8w+ star, I have been receiving feedback that the basic part (the content of the first 15 days) is difficult for novices, and it is recommended to have a supporting video for explanation. Recently, I recreated a project called “Python-Core-50-Courses” 123 from the basic part of the content . This part of the content was rewritten in a simpler and more popular way and accompanied by a video explanation . Beginners can pay attention to this new project. If domestic users are slow to access GitHub, they can also pay attention to the “Learn Python from scratch” 39 column on my Zhihu Python-Jack 15 . The column will continue to be updated, and the content of “data analysis” that everyone is looking forward to will also be coming soon Online, welcome everyone to pay attention to my columns, articles and answers in Zhihu .
The creation is not easy. Thank you for your rewards and support. The money will basically not be used to buy coffee. Instead, it will be donated to people in need through platforms such as Tencent Charity, Meituan Charity, and Didi Chuan ( click to 3 learn about donations). Those who need to join the QQ exchange group can scan the QR code below. The exchange group will provide you with learning resources and answers to questions , and will continue to bring you free online Python experience courses and industry open courses , so stay tuned.
Analysis of Python application fields and employment situation
Simply put, Python is an “elegant”, “clear”, and “simple” programming language.
- Low learning curve, even non-professionals can get started
- Open source system, with a strong ecosystem
- Interpreted language, perfect platform portability
- Support object-oriented and functional programming
- Ability to extend functions by calling C/C++ code
- High degree of code standardization and strong readability
Python has its place in several popular fields.
- Cloud Infrastructure-Python / Java / Go
- DevOps-Python / Shell / Ruby / Go
- Web crawler-Python / PHP / C++
- Data analysis and mining-Python / R / Scala / Matlab
- Machine Learning-Python / R / Java / Lisp
As a Python developer, the main areas of employment include:
- Python server background development / game server development / data interface development engineer
- Python Automation Operation and Maintenance Engineer
- Python data analysis/ data visualization/ big data engineer
- Python crawler engineer
- Python chatbot development / image recognition and visual algorithms / deep learning engineer
The figure below shows the Python recruitment demand and salary ranking list in major cities (as of May 2018).
A few suggestions for beginners:
- Make English as your working language. (Make English as your working language)
- Practice makes perfect. (Practice makes perfect)
- All experience comes from mistakes. (All experience comes from mistakes.)
- Don’t be one of the leeches. (Don’t be one of the leeches.)
- Either outstanding or out. (either outstanding or out)
Day01~15- Python language foundation 207
Day01-Getting to Know Python 133
- Introduction to Python-History of Python / Advantages and Disadvantages of Python / Application Fields of Python
- Set up a programming environment-Windows environment / Linux environment / MacOS environment
- Run Python program from terminal-Hello, world / print function / run program
- Use IDLE-Interactive Environment (REPL) / write multiple lines of code / run the program / exit IDLE
- Comment-The role of comments / single-line comments / multi-line comments
Day02- Language Elements 53
- Programs and bases-instructions and programs / Von Neumann machine / Binary and decimal / Octal and hexadecimal
- Variables and types-variable naming / variable use / input function / check variable type / type conversion
- Numbers and strings-integer / floating point / complex number / string / string basic operation / character encoding
- Operators-Mathematical Operators / Assignment Operators / Comparison Operators / Logical Operators / Identity Operators / Priority of Operators
- Application case-Convert Fahrenheit temperature to Celsius temperature / Enter the radius of the circle to calculate the circumference and area / Enter the year to determine whether it is a leap year
Day03- branch structure 25
- Application scenarios of branch structure-condition / indentation / code block / flow chart
- if statement-simple if / if-else structure / if-elif-else structure / nested if
- Application case-User authentication / Interchange of imperial units and metric units / Decide what to do by rolling the dice / Converting percentile scores to grade system / Evaluation of piecewise function / Enter the length of three sides, if they can form a triangle, calculate the perimeter and area
Day04- Loop structure 25
- Application scenarios of loop structure-condition/indentation/code block/flow chart
- while loop-basic structure / break statement / continue statement
- for loop-basic structure / range type / branch structure in loop / nested loop / end program early
- Application case-1~100 summation / judging prime numbers / guessing the number game / printing the nine-nine table / printing the triangle pattern / monkeys eating peaches / hundred money and hundred chickens
Day05- Constructing program logic 18
- Classic case: the number of daffodils / a hundred money and a hundred chickens / Craps gambling game
- Practice questions: Fibonacci sequence / perfect numbers / prime numbers
Day06-Use of functions and modules 22
- The role of the function-bad smell of the code / function module encapsulation
- Define function-def statement/function name/parameter list/return statement/call custom function
- Call function-Python built-in functions / import modules and functions
- Function parameters-default parameters / variable parameters / keyword parameters / named keyword parameters
- The return value of the function-no return value / return a single value / return multiple values
- Scope issues-local scope / nested scope / global scope / built-in scope / scope-related keywords
- Manage functions with modules-the concept of modules / use custom modules to manage functions / what happens when naming conflicts (the same module and different modules)
Day07- Strings and common data structures 18
- The use of string-calculation length / subscript operation / slicing / common methods
- Basic usage of list-define list / use the following table to access elements / subscript out of range / add element / delete element / modify element / slice / loop traversal
- Common List Operations-Concatenation / Copy (Copy Elements and Copy Array) / Length / Sort / Reverse / Find
- Generate list-use range to create a list of numbers / generate expressions / generators
- Use of tuples-define tuples / use values in tuples / modify tuple variables / tuples and list conversion
- Basic usage of collection-the difference between collection and list / create collection / add element / delete element / empty
- Common Set Operations-Intersection / Union / Difference / Symmetric Difference / Subset / Superset
- The basic usage of the dictionary-the characteristics of the dictionary / create a dictionary / add elements / delete elements / take values / empty
- Common dictionary operations-keys() method/ values() method/ items() method/ setdefault() method
- Basic exercises-marquee effect / find the largest element in a list / statistical test score average / Fibonacci sequence / Yang Hui triangle
- Comprehensive case-Two-color ball number selection / Tic-Tac-Toe
Day08- Fundamentals of Object 14 - Oriented Programming 14
- Classes and objects-what is a class / what is an object / other related object-oriented concepts
- Define the class-basic structure / attributes and methods / constructor / destructor / str method
- Use object-create object / send message to object
- Four pillars of object-oriented-abstraction / encapsulation / inheritance / polymorphism
- Basic exercises-define student class/ define clock class/ define graphics class/ define car class
Day09- Advanced Object Oriented 12
- Attribute-class attribute / instance attribute / attribute accessor / attribute modifier / attribute deleter / use slots
- Method in class-instance method/class method/static method
- Operator overloading-add / sub / or /getitem / setitem / len / repr / gt / lt / le / ge / eq / ne / contains
- The relationship between classes (of objects)-association/inheritance/dependency
- Inheritance and polymorphism-what is the inheritance/ syntax of inheritance/ calling parent method/ method rewriting/ type determination/ multiple inheritance/ diamond inheritance (diamond inheritance) and C3 algorithm
- Comprehensive Case-Salary Settlement System / Automatic Book Discount System / Customized Score
Day10- Graphical user interface and game development 8
- Use tkinter to develop GUI programs
- Use pygame third party library to develop game applications
- “Big ball eats small ball” game
Day11- files and exceptions 4
- Read file-read the entire file / line by line / file path
- Write file-overwrite writing/additional writing/text file/binary file
- Exception Handling-The importance of exception mechanism / try-except code block / else code block / finally code block / built-in exception type / exception stack / raise statement
- Data Persistence-Overview of CSV File / Application of csv Module / JSON Data Format / Application of json Module
Day12- strings and regular expressions 4
- Advanced string operations-escape characters / original strings / multi-line strings / in and not in operators / methods at the beginning of is / join and split methods / strip related methods / pyperclip module / invariant strings and variable characters Use of String/StringIO
- Introduction to regular expressions-the role of regular expressions / metacharacters / escapes / quantifiers / grouping / zero-width assertions / greedy matching and lazy matching laziness / using the re module to implement regular expression operations (matching, searching, replacing, capturing)
- Use regular expressions-re module / compile function / group and groups method / match method / search method / findall and finditer method / sub and subn method / split method
- Application Case-Use regular expressions to verify the input string
Day13- Processes and threads 7
- The concept of processes and threads-what is a process / what is a thread / multi-threaded application scenarios
- Use process-fork function / multiprocessing module / process pool / inter-process communication
- Use thread-thread module / threading module / Thread class / Lock class / Condition class / thread pool
Day14- Introduction to Network Programming and Network Application Development 12
- Computer Network Foundation-History of Computer Network Development / “TCP-IP” Model / IP Address / Port / Protocol / Other Related Concepts
- Network Application Mode-“Client-Server” Mode / “Browser-Server” Mode
- Access network resources based on HTTP protocol-Network API overview / access URL / requests module / parse JSON format data
- Python network programming-the concept of socket / socket module / socket function / create TCP server / create TCP client / create UDP server / create UDP client / SocketServer module
- Email-SMTP protocol / POP3 protocol / IMAP protocol / smtplib module / poplib module / imaplib module
- SMS Service-Call SMS Service Gateway
Day15- Image and document processing 7
- Use Pillow to process pictures-picture reading and writing / picture synthesis / geometric transformation / color conversion / filter effect
- Reading and writing Word documents-text content processing/paragraph/header and footer/style processing
- Read and write Excel files-xlrd module / xlwt module
- Generate PDF files-pypdf2 module / reportlab module
Day16~Day20- Advanced Python language 9
- Common data structure
- Advanced usage of functions-“first-class citizens” / higher-order functions / Lambda functions / scopes and closures / decorators
- Advanced Object-Oriented Knowledge-“Three Pillars” / Relationship Between Classes / Garbage Collection / Magic Properties and Methods / Mixing / Metaclasses / Object-Oriented Design Principles / GoF Design Pattern
- Iterators and generators-related magic methods / two ways to create generators /
- Concurrent and asynchronous programming-multithreading/multiprocessing/asynchronous IO/async and await
Day21~30-Introduction to Web Front End 8
- Use HTML tags to carry page content
- Render the page with CSS
- Handling interactive behaviors with JavaScript
- jQuery entry and improvement
- Getting started with Vue.js
- Use of Element
- Use of Bootstrap
Day31~35- Fun with Linux operating system 6
- Operating system development history and Linux overview
- Linux basic commands
- Utilities in Linux
- Linux file system
- Vim editor application
- Environment variables and Shell programming
- Software installation and service configuration
- Network access and management
- Other related content
Day36~40- Database basics and advanced 5
- Relational database MySQL 1
- Overview of relational databases
- MySQL installation and use
- Use of SQL
- DDL-Data Definition Language-create / drop / alter
- DML-Data Manipulation Language-insert / delete / update / select
- DCL-Data Control Language-grant / revoke
- related information
- Paradigm Theory-the guiding ideology for designing two-dimensional tables
- Data integrity
- Data consistency
- Operate MySQL in Python
- Getting started with NoSQL 1
- NoSQL overview
- Redis overview
- Mongo overview
Day41~55- Practical Django 11
Day41- Django quick start 7
- Web application working mechanism
- HTTP request and response
- Django framework overview
- Quick start in 5 minutes
Day42- in 3 - depth model 3
- Relational database configuration
- Use ORM to complete CRUD operations on the model
- Use of management background
- Django model best practices
- Model definition reference
Day43- Static resources and Ajax requests
- Load static resources
- Ajax overview
- Implement voting function with Ajax
Day44- Cookie and Session 3
- Implement user tracking
- The relationship between cookie and session
- Django framework support for session
- Cookie read and write operations in view functions
Day45- reports and logs 3
- Modify the response header through HttpResponse
- Use StreamingHttpResponse to process large files
- Use xlwt to generate Excel reports
- Use reportlab to generate PDF reports
- Use ECharts to generate front-end charts
Day46- Log and debug toolbar 3
- Configuration log
- Configure Django-Debug-Toolbar
- Optimize ORM code
Day47- Middleware Application 2
- What is middleware
- Middleware built into Django framework
- Custom middleware and its application scenarios
Day48- Introduction to Front-end and 2 Back- end Separation 2
- Return data in JSON format
- Render the page with Vue.js
Day49- Introduction to RESTful architecture and DRF 3Day50- RESTful architecture and advanced DRF 1Day51- Use cache 2
- The first law of website optimization
- Use Redis to provide caching service in Django project
- Read and write cache in view function
- Use decorators to implement page caching
- Provide caching service for data interface
Day52- file upload 2
- File upload form control and image file preview
- How the server handles uploaded files
Day53- Asynchronous tasks and timing tasks 2
- The second law of website optimization
- Configure message queue service
- Use celery to implement task asynchronization in the project
- Use celery to implement timing tasks in the project
Day54- unit testing 2Day55- project online 3
- Unit testing in Python
- Django framework’s support for unit testing
- Use version control system
- Configure and use uWSGI
- Dynamic separation and Nginx configuration
- Configure HTTPS
- Configure domain name resolution
Day56~60- Actual Combat Flask 1
Day56- Getting started with Flask 2Day57- Use of templates 3Day58- Form processing 2Day59- database operations 1Day60- actual project 1
Day61~65- Actual Tornado 1
Day61- Preliminary Knowledge
- Concurrent programming
- I/O mode and event driven
Day62- Getting started with Tornado 1
- Tornado overview
- 5 minutes to get started with Tornado
- Route resolution
- Request handler
Day63- Asynchronization 2
- The use of aiomysql and aioredis
Day64- Application of WebSocket 6
- Introduction to WebSocket
- WebSocket server-side programming
- WebSocket client programming
- Project: Web chat room
Day65- Project Actual Combat 3
- Front-end and back-end separate development and interface document writing
- Use Vue.js to achieve front-end rendering
- Use ECharts to realize report function
- Use WebSocket to implement push service
Day66~75- crawler development 3
Day66- Web crawlers and related tools 4
- The concept of web crawler and its application field
- Discussion on the legality of web crawlers
- Tools for developing web crawlers
- The composition of a crawler
Day67- Data Collection and Analysis 3
- Standards and tripartite libraries for data collection
- Three ways of page parsing: regular expression parsing/XPath parsing/CSS selector parsing
Day68- store data
- How to store massive amounts of data
- Implement data caching
Day69- concurrent download 3
- Multithreading and multiprocessing
- Asynchronous I/O and coroutines
- The use of async and await keywords
- Application of the three party library aiohttp
Day70- Parse dynamic content 2
- JavaScript reverse engineering
- Use Selenium to get dynamic content
Day71- Form interaction and verification code processing 1
- Auto submit form
- Cookie pool application
- Verification code processing
Day72- Getting Started with Scrapy 5
- Scrapy crawler framework overview
- Install and use Scrapy
Day73- Scrapy Advanced Application 1
- Spider usage
- Middleware application: download middleware / spider middleware
- Scrapy docks with Selenium to capture dynamic content
- Scrapy deployed to Docker
Day74- Scrapy distributed implementation 5
- Principles of distributed crawlers
- Scrapy distributed implementation
- Use Scrapyd to implement distributed deployment
Day75- Reptile Project Actual Combat 2
- Crawling recruitment website data
- Crawling real estate industry data
- Crawling used car trading platform data
Day76~90- Data Analysis and Machine Learning 7
Reminder : The content of data analysis and machine learning is in the code folder. It is the code and notes written in jupyter notebook. You need to start jupyter notebook and then open the corresponding file for learning. Relevant documents will continue to be supplemented in 2020, and I hope you will continue to pay attention.
Day76- Machine Learning Fundamentals 9Day77- Application of Pandas 3Day78-Application of NumPy and SciPy 1Day79- Matplotlib and data visualization 1Day80- k nearest neighbor (KNN) classification 1Day81- Decision Tree 4Day82- Bayesian ClassificationDay83- Support Vector Machine (SVM) 1Day84- K-means clustering 5Day85- regression analysis 1Day86- Introduction to Big Data Analysis 4Day87- Advanced Big Data Analysis 3Day88- Getting Started with Tensorflow 1Day89- Tensorflow combat 3Day90- Recommended system combat 1
Day91~100- Team project development 3
Day 91: Problems and solutions for team project development 4
- Software process model
- Classic process model (waterfall model)
- Feasibility analysis (whether the research is done or not), and output the “feasibility analysis report”.
- Requirement analysis (study what to do), output “requirements specification” and product interface prototype diagram.
- Outline design and detailed design, output conceptual model diagram (ER diagram), physical model diagram, class diagram, sequence diagram, etc.
- Coding/testing.
- Go online/maintenance.The biggest disadvantage of the waterfall model is that it cannot embrace changes in requirements. The product can only be seen after the entire process is over, and the team morale is low.
- Agile Development (Scrum)-Product Owner, Scrum Master, R&D Staff-Sprint
- Product Backlog (user story, product prototype).
- Planning meetings (evaluation and budgeting).
- Daily development (stand-up meeting, Pomodoro Technique, pair programming, test first, code refactoring…).
- Fix bugs (problem description, reproduction steps, testers, designees).
- release version.
- Review meeting (Showcase, users need to participate).
- Retrospective meeting (a summary of the current iteration cycle).
Supplement: Manifesto for Agile Software Development
* **Individuals and interactions are** higher than processes and tools * **Working software is** higher than detailed documentation * **Customer cooperation is** higher than contract negotiation * **Responding to changes is** better than following a plan
Roles: product owner (the person who decides what to do and can approve the demand), team leader (solve various problems, focus on how to work better, shield the external influence on the development team), development team (project executive, Specifically refers to developers and testers).
Preparation work: business case and capital, contract, vision, initial product requirements, initial release plan, shareholding, and team formation.
Agile teams usually have 8-10 people.
Estimation of workload: Quantify the development tasks, including prototype, logo design, UI design, front-end development, etc., try to break down each work to the minimum amount of tasks, the minimum task amount standard is that the working time cannot exceed two days, and then estimate the overall project time . Post each task on the Kanban board, which is divided into three parts: to do (to be completed), in progress (in progress) and done (completed).
- Project team formation
- Team composition and roles
Note: Thank you Ms. Fu Xiangying for helping me draw the following beautiful company organization chart.
- Programming specification and code review (flake8, pylint) 1
- Some “conventions” in Python (please refer to “Python Conventions-How to Write Pythonic Code” )
- Reasons affecting code readability:
- Code comments are too few or no comments
- Code breaks language best practices
- Anti-pattern programming (spaghetti code, copy-paste programming, conceit programming,…)
- Introduction to team development tools
- Version control: Git, Mercury
- Defect management: Gitlab , Redmine
- Agile closed-loop tools: Zen Tao , JIRA
- Continuous integration: Jenkins , Travis-CIPlease refer to “Problems and Solutions for Team Project Development” 4 .
Project topic selection and business understanding
- Topic selection range setting
- CMS (user terminal): news aggregation website, Q&A/sharing community, film review/book review website, etc.
- MIS (user side + management side): KMS, KPI assessment system, HRS, CRM system, supply chain system, warehouse management system, etc.
- App background (management terminal + data interface): second-hand trading, newspapers and magazines, niche e-commerce, news information, travel, social, reading, etc.
- Other types: own industry background and work experience, business is easy to understand and control.
- Requirements understanding, module division and task allocation
- Need understanding: brainstorming and competitive product analysis.
- Module division: draw a mind map (XMind), each module is a branch node, and each specific function is a leaf node (expressed in verbs). It is necessary to ensure that each leaf node cannot regenerate a new node, and determine each leaf The importance, priority and workload of the node.
- Task assignment: The project leader assigns tasks to each team member according to the above indicators. 1
- Develop project schedule (updated daily)
ModuleFeaturespersonnelstatuscarry outWorking hoursPlan startedActual startEnd of planActual endRemarks
comment | add comment | Wang Dahui | Ongoing | 50% | 4 | 2018/8/7 | 2018/8/7 | |||
Delete comment | Wang Dahui | wait | 0% | 2 | 2018/8/7 | 2018/8/7 | ||||
View comments | Bai Yuanfang | Ongoing | 20% | 4 | 2018/8/7 | 2018/8/7 | Code review required | |||
Comment vote | Bai Yuanfang | wait | 0% | 4 | 2018/8/8 | 2018/8/8 |
- OOAD and database design
- UML (Unified Modeling Language) class diagram 2
- Create table from model (forward engineering)
python manage.py makemigrations app python manage.py migrate
- Use PowerDesigner to draw physical model diagrams. 1
- Create model from data table (reverse engineering)
python manage.py inspectdb > app/models.py
Day 92: Docker container explained 4
- Introduction to Docker
- Install Docker
- Use Docker to create containers (Nginx, MySQL, Redis, Gitlab, Jenkins)
- Build Docker image (write Dockerfile and related instructions)
- Container Orchestration (Docker-compose)
- Cluster Management (Kubernetes)
Day 93: MySQL performance optimization 6Day 94: Web API interface design 3Day 95: [Develop commercial projects using Django](./Day91-100/95. Use Django to develop commercial projects.md)Public issues in project development
- Database configuration (multi-database, master-slave replication, database routing)
- Cache configuration (partition cache, key settings, timeout settings, master-slave replication, failure recovery (sentinel))
- Log configuration
- Analysis and debugging (Django-Debug-ToolBar)
- Useful Python modules (date calculation, image processing, data encryption, three-party API)
REST API design
- RESTful architecture
- API interface document writing
- django-REST-framework application
Analysis of key and difficult points in the project
- Use cache to relieve database pressure-Redis
- Use message queues for decoupling and peak clipping-Celery + RabbitMQ
Day 96: Software testing and automated testing 11unit test
- Type of test
- Write unit tests (unittest, pytest, nose2, tox, ddt,…)
- Test coverage (coverage)
Project deployment
- Preparation before deployment
- Key settings (SECRET_KEY / DEBUG / ALLOWED_HOSTS / cache / database)
- HTTPS / CSRF_COOKIE_SECUR / SESSION_COOKIE_SECURE
- Log related configuration
- Review of Linux commonly used commands
- Installation and configuration of Linux common services
- Use of uWSGI/Gunicorn and Nginx
- Comparison of Gunicorn and uWSGI
- For simple applications that do not require a lot of customization, Gunicorn is a good choice. The learning curve of uWSGI is much steeper than that of Gunicorn. Gunicorn’s default parameters can already be adapted to most applications.
- uWSGI supports heterogeneous deployment.
- Since Nginx itself supports uWSGI, Nginx and uWSGI are generally bundled and deployed online, and uWSGI is a fully functional and highly customized WSGI middleware.
- In terms of performance, Gunicorn and uWSGI are actually comparable.
- Use virtualization technology (Docker) to deploy test and production environments
Performance Testing
- Use of AB
- Use of SQLslap
- Use of sysbench
automated test
- Use Shell and Python for automated testing
- Use Selenium to implement automated testing
- Selenium IDE
- Selenium WebDriver
- Selenium Remote Control
- Introduction of testing tool Robot Framework
Day 97: Analysis of technical points of e-commerce website 9Day 98: Project deployment and performance tuning 11
- MySQL database tuning
- Web server performance optimization
- Nginx load balancing configuration
- Keepalived achieves high availability
- Code performance tuning
- Multithreading
- Asynchronous
- Static resource access optimization
- Cloud storage
- CDN
Day 99: Public questions in the interview 16Day 100: Python interview question set 32
'Study > 개발 Tip' 카테고리의 다른 글
[개발 Tip] 모든 프로그래밍 관련 질문을 해결할 수 있는 10 개의 사이트 (0) | 2021.05.30 |
---|---|
[Firebase 연동] SHA 인증 지문 얻기 (0) | 2021.05.27 |
[개발 TIP] 무료 호스팅 | 무료 SSL 얻기 (0) | 2021.05.03 |
[개발 Tip] HTML로 할 수 있는 것들은 무엇이 있을까? (0) | 2021.05.02 |
[개발 Tip] 웹에서 OS 구동하기 (0) | 2021.05.02 |