SSo

http://khanrc.tistory.com/entry/%ED%95%9C%EA%B8%80-in-the-dictionary-feat-pretty

파싱하다가 딕셔너리에서 한글 인코딩이 막혀서 찾음

일반적으로 python에서 모듈을 import할때 일반적으로 시작부분에 선언 후 코드를 작성한다.

그런데 모듈을 처음에 import하는게 아니라 상황에 맞게 동적으로 모듈을 import할 수 있다.

방법은 importlib 패키지를 사용하거나 __import__()함수를 사용하면 된다.

사실 둘의 차이는 별로 없다.

일단 importlib은 import구문(statement)을 수행하는 하나의 package일 뿐이며, __import__()함수는 import구문을 구성하는 expression이기 때문이다.

즉 __import__()함수는 importlib보다 low level로 호출하는 것 뿐이다.

importlib

import하고 싶은 module명을 module_name이라는 변수로 받았다고 하자.

이런 경우 importlib을 이용해 아래처럼 간단히 이용할 수 있다.

import importlib

def load_module_func(module_name):
	mod = importlib.import_module(module_name)
	return mod

mod란 변수를 통해 module 안에 class를 호출하는 것이 가능하다.

__import__()

importlib 보다 좀 더 low level 로 __import__ 를 이용해서도 구현이 가능하다. 물론 이걸 사용하면, importlib 패키지 자체를 import 하는 과정은 생략해도 된다. __import__() 는 built-in 함수이기 때문에 그냥 사용하면 된다. 

def load_module_func(module_name):
	mod = __import__('%s' %(module_name), fromlist=[module_name])
	return mod 

코드게이트 문제를 풀다가 알게 된 라이브러리다.

수식을 주면 변수를 알아내도록 도와주는 도구임.

설치는 아래 링크를 참고

https://github.com/Z3Prover/z3

웹 문제를 풀다보면 기본적인 치환암호를 푸는 문제가 종종 나온다.

그래서 단순 문자에 대한 치환암호를 복호화 하는 코드를 짰다.

문자를 1대1 대응시켜서 변환시키는 파이썬 라이브러리도 있는데 그건 여기를 참고하자.

우선 아래는 암호화된 문자열을 받아서 모든 경우의 수를 계산해 출력해주는 코드다.

def decode(strings):
        for i in range(0, 26):
                key = ""
                for string in strings:
                        ascii_str = ord(string)
                        if (ascii_str > 64 and ascii_str < 91):
                                if (ascii_str+i) > 90:
                                        ret_val = 64 + ((ascii_str+i)-90)
                                else:
                                        ret_val = ascii_str + i
                        elif (ascii_str > 96 and ascii_str < 123):
                                if (ascii_str+i) > 122:
                                        ret_val = 96 + ((ascii_str+i)-122)
                                else:
                                        ret_val = ascii_str + i
                        else:
                                ret_val = ascii_str
                        key += chr(ret_val)
                print "[+] {} : {}".format(i, key)

def main():
        incoded_flag = "ABCDE" # 치환된 문자열
        decode(incoded_flag)

if __name__ == "__main__":
        main()

try .. finally

try문에는 finally절을 사용할 수 있다. finally절은 try문 수행 도중 예외 발생 여부에 상관없이 항상 수행된다. 보통 finally절은 사용한 리소스를 close해야 할 경우에 많이 사용된다.

다음의 예를 보자.

f = open('foo.txt', 'w')
try:
    # 무언가를 수행한다.
finally:
    f.close()

foo.txt라는 파일을 쓰기 모드로 연 후에 try문이 수행된 후 예외 발생 여부에 상관없이 finally절에서 f.close()로 열린 파일을 닫을 수 있다.

Ref : https://wikidocs.net/30

This is an updated version of my previous blog post on few recommended practices for optimizing your Python code.

A codebase that follows best practices is highly appreciated in today's world. It's an appealing way to engage awesome developers if your project is Open Source. As a developer, you want to write efficient and optimized code, which is:

Code that takes up minimum possible memory, executes faster, looks clean, is properly documented, follows standard style guidelines, and is easy to understand for a new developer.

The practices discussed here might help you contribute to an Open Source organization, submit a solution to an Online Judge, work on large data processing problems using machine learning, or develop your own project.

Practice 1: Try Not To Blow Off Memory!

A simple Python program may not cause many problems when it comes to memory, but memory utilization becomes critical on high memory consuming projects. It's always advisable to keep memory utilization in mind from the very beginning when working on a big project.

Unlike in C/C++, Python’s interpreter performs the memory management and users have no control over it. Memory management in Python involves a private heap that contains all Python objects and data structures.

The Python memory manager internally ensures the management of this private heap. When you create an object, the Python Virtual Machine handles the memory needed and decides where it'll be placed in the memory layout.

However, greater insight into how things work and different ways to do things can help you minimize your program's memory usage.

• Use generators to calculate large sets of results:

Generators give you lazy evaluation. You use them by iterating over them: either explicitly with 'for' or implicitly, by passing it to any function or construct that iterates.

You can think of generators returning multiple items like they're returning a list — instead of returning them all at once, however, they return them one-by-one. The generator function is paused until the next item is requested. Read more about Python Generators here.

• For large numbers/data crunching, you can use libraries like Numpy, which gracefully handles memory management.

• Use format instead of + for generating strings — In Python, str is immutable, so the left and right strings have to be copied into the new string for every pair of concatenations.
  If you concatenate four strings of length 10, you'll be copying (10+10) + ((10+10)+10) + (((10+10)+10)+10) = 90 characters instead of just 40 characters. Things get quadratically worse as the number and size of the string increases.
  Java optimizes this case by transforming the series of concatenations to use StringBuilder some of the time , but CPython doesn't. Therefore, it's advised to use .format or % syntax. If you can't choose between .format and %, check out this interesting StackOverflow thread.

• Use slots when defining a Python class. You can tell Python not to use a dynamic dict, and only allocate space for a fixed set of attributes, eliminating the overhead of using one dict for every object by setting __slots__ on the class to a fixed list of attribute names. Read more about slots here.

• You can track your memory usage at object level by using built-in modules like resource and objgraph.

• Managing memory leaks in Python can be a tough job, but luckily there are tools like heapy for debugging memory leaks. Heapy can be used along with objgraph to watch allocation growth of diff objects over time. Heapy can show which objects are holding the most memory. Objgraph can help you find back-references to understand exactly why they cannot be freed. You can read more about diagnosing memory leaks in Python here.

You can read in detail about Python memory management by the developers of Theano here.

Practice 2: Python2 or Python3?

When starting a new Python project, or even just learning Python, you might find yourself with the dilemma of choosing between Python2 or Python3. This is a widely discussed topic with many opinions and good explanations on the Internet.

On one hand, Python3 has some great new features. On the other hand, you may want to use a package that only support Python2, and Python3 is not backward-compatible. This means that running your Python2 code on a Python3.x interpreter can possibly throw errors.

However, it is possible to write code in a way that works on both Python2 and Python3 interpreters. The most common way is to use packages like _future, builtins, and six to maintain a single, clean Python3.x compatible codebase that supports both Python2 and Python3 with minimal overhead.

python-future is the missing compatibility layer between Python2 and Python3. It provides future and past packages with backports and forward ports with features from Python3 and Python2. It also comes with futurize and pasteurize, customized 2-to-3 based scripts that help you easily convert either Py2 or Py3 code to support both Python2 and Python3 in a single clean Py3-style codebase, module by module.

Please check out the excellent Cheat Sheet for writing Python 2-3 compatible code by Ed Schofield. If you're more into watching videos than reading, you may find his talk at PyCon AU 2014, “Writing 2/3 compatible code” helpful.

Practice 3: Write Beautiful Code because - "The first impression is the last impression."

Sharing code is a rewarding endeavor. Whatever the motivation, your good intentions may not have the desired outcome if people find your code hard to use or understand. Almost every organization follows style guidelines that developers have to follow for consistency, easy debugging, and ease of collaboration. The Zen of Python is like a mini style and design guide for Python. Popular style guidelines for Python include:

1. PEP-8 style guide
2. Python Idioms and efficiency
3. Google Python Style Guide

These guidelines discuss how to use: whitespace, commas, and braces, the object naming guidelines, etc. While they may conflict in some situations, they all have the same objective — "Clean, Readable, and Debuggable standards for code."

Stick to one guide, or follow your own, but don't try to follow something drastically different from widely accepted standards.

Using static code analysis tools

There are lots of open source tools available that you can use to make your code compliant with standard style guidelines and best practices for writing code.

Pylint is a Python tool that checks a module for coding standards. Pylint can be a quick and easy way of seeing if your code has captured the essence of PEP-8 and is, therefore, ‘friendly’ to other potential users.

It also provides you with reports with insightful metrics and statistics that may help you judge code quality. You can also customize it by creating your own .pylintrc file and using it.

Pylint is not the only option — there are other tools like PyChecker, PyFlakes, and packages like pep8 and flakes8.
My recommendation would be to use coala, a unified static code analysis framework that aims to provide language agnostic code analysis via a single framework. Coala supports all the linting tools I mentioned previously, and is highly customizable.

Documenting the code properly

This aspect is most critical to the usability and readablity of your codebase. It is always advised to document your code as extensively as possible, so that other developers face less friction to understand your code.
A typical inline-documentation of a function should include:

• A one line summary of what the function does.
• Interactive examples, if applicable. These could be referred by the new developer to quickly observe the usage and expected output of your function. As well as you can use the doctest module to assert the correctness of these examples (running as tests). See the doctest documentation for examples.
• Parameters documentation (generally one line describing the parameter and its role in the function)
• Return type documentation (unless your function doesn't return anything!)

Sphinx is a widely used tool for generating and managing your project documentation. It offers a lots of handy features that would reduce your efforts in writing a standard documentation. Moreover, you can publish your documentation at Read the Docs for free, which is the most common way of hosting documentation for projects.
The Hitchiker's guide to Python for documentation contains some interesting information that may be useful to you while documenting your code.

Practice 4: Speed Up Your Performance

Multiprocess, not Multi-thread

When it comes to improving the execution time of your multiple-task code, you may want to utilize multiple cores in the CPU to execute several tasks simultaneously. It may seem intuitive to spawn several threads and let them execute concurrently, but, because of the Global Interpreter Lock in Python, all you're doing is making your threads execute on the same core turn by turn.

To achieve actual parallelization in Python, you might have to use a Python multiprocessing module. Another solution might be outsourcing the tasks to:

1. The operating system (by doing multi-processing)
2. Some external application that calls your Python code (e.g., Spark or Hadoop)
3. Code that your Python code calls (e.g. you could have your Python code call a C function that does the expensive multi-threaded stuff).

Apart from multiprogramming, there are other ways to boost your performance. Some of them include:

• Using the latest version of Python: This is the most straightforward way because new updates generally include enhancements to already existing functionalities in terms of performance.

• Use built-in functions wherever possible: This also aligns with the DRY principle — built-in functions are carefully designed and reviewed by some of the best Python developers in the world, so they're often the best way to go.

• Consider using Ctypes: Ctypes provides an interface to call C shared functions from your Python code. C is a language closer to machine level, which makes your code execute much faster compared to similar implementations in Python.

• Using Cython: Cython is a superset Python language that allows users to call C functions and have static type declarations, which eventually leads to a simpler final code that will probably execute much faster.

• Using PyPy: PyPy is another Python implementation that has a JIT (just-in-time) compiler, which could make your code execution faster. Though I've never tried PyPy, it also claims to reduce your programs' memory consumption. Companies like Quora actually use PyPy in production.

• Design and Data Structures: This applies to every language. Make sure you're using the right data structures for your purpose, declare variables at the right place, wisely make use of identifier scope, and cache your results wherever it makes sense, etc.

A language specific example that I could give is — Python is usually slow with accessing global variables and resolving function addresses, so it's faster to assign them to a local variable in your scope and then access them.

Practice 5: Analyzing your code

It's often helpful to analyze your code for coverage, quality, and performance. Python comes with the cProfile module to help evaluate performance. It not only gives the total running time, it also times each function separately.

It then tells you how many times each function was called, which makes it easy to determine where you should make optimizations. Here's what a sample analysis by cProfile looks like:

• memory_profiler is a Python module for monitoring memory consumption of processes, as well as a line-by-line analysis of memory consumption for Python programs.

• objgraph allows you to show the top N objects occupying our Python program’s memory, what objects have been deleted or added over a period of time, and all references to a given object in your script.

• resource provides basic mechanisms for measuring and controlling system resources utilized by a program. The module's two prime uses include limiting the allocation of resources and getting information about the resource's current usage.

Practice 6: Testing and Continuous Integration

Testing:

It is good practice to write unit tests. If you think that writing tests aren't worth the effort, take a look at this StackOverflow thread. It's better to write your tests before or during coding. Python provides unittest modules to write unit tests for your functions and classes. There are frameworks like:

• nose - can run unittest tests and has less boilerplate.
• pytest - also runs unittest tests, has less boilerplate, better reporting, and lots of cool, extra features.
  To get a good comparison among these, read the introduction here.

Not to forget the doctest module, which tests your source code using the interactive examples illustrated in the inline documentation.

Measuring coverage:
Coverage is a tool for measuring Python program code coverage. It monitors your program, notes which parts of the code have been executed, then analyzes the source to identify code that could've been executed but was not.

Coverage measurement is typically used to gauge the effectiveness of tests. It can show which parts of your code are being exercised by tests, and which are not. It is often advisable to have 100% branch coverage, meaning your tests should be able to execute and verify the output of every branch of the project.

Continuous Integration:
Having a CI system for your project from the very beginning can be very useful for your project in the long run. You can easily test various aspects of your codebase using a CI service. Some typical checks in CI include:

• Running tests in a real world environment. There are cases when tests pass on some architectures and fail on others. A CI service can let you run your tests on different system architectures.

• Enforcing coverage constraints on your codebase.

• Building and deploying your code to production (you can do this across different platforms)

There are several CI services available nowadays. Some of the most popular ones are Travis, Circle (for OSX and Linux) and Appveyor (for Windows). Newer ones like Semaphore CI also seem reliable, per my initial use. Gitlab (another Git repository management platform like Github) also supports CI, though you'll need to configure it explicitly, like with other services.

Update: This post was entirely based on my personal experiences. There may be a lot of things which I missed (or I'm not aware of). If you have something interesting to share, do let me know in comments. Someone started a thread on the same topic in HN, I'd recommend you to check it out https://news.ycombinator.com/item?id=15046641 for more critical discussions regarding this post. I'll try to address all the suggestions and keep updating this post frequently.

그리고 아래는 구글의 python 작성 스타일 가이드 문서다.

https://google.github.io/styleguide/pyguide.html

Ref.https://www.codementor.io/satwikkansal/python-practices-for-efficient-code-performance-memory-and-usability-aze6oiq65

리스트안의 중복된 값의 개수

w_count = {} 
lists = ["가나다라", "BYE", "HELLO", "123", "Asd1", "123", "HELLO", "bye", "BYE"]
for lst in lists:
    try: w_count[lst] += 1
    except: w_count[lst]=1
print w_count

>>reulst

{'bye': 1, 'Asd1': 1, '가나다라': 1, '123': 2, 'BYE': 2, 'HELLO': 2}